Shadowing is when you have two separate variables with the same identifier. It is beyond obvious that all the x's refer to the same variable in dilap's example. Contrast that with an actual example of shadowing[0], in which it is clear that the same identifier is being used to refer to two different variables.

[0]: https://en.wikipedia.org/wiki/Variable_shadowing#Python

I'm using shadowing in the rust sense, not the python sense.

They are the exact same sense!

The exact same idea of shadowing is also present in, for example, sentences in first order logic.

Setting aside any terminology for a second, consider this rust program:

  fn main() {
   let x = 1;
   let capture = || x;
   let x = 2;
   println!("{}", capture());
   println!("{}", x)
  }

This will print 1 and then 2, whereas python would print 2 and 2.

Hence, you can see that the formulation "let mut" is equivalent to python, not "let" followed by "let".

Here's the rust program that prints 2 and 2:

  fn main() {
   let mut x = 1;
   let ptr = &x as *const i32;
   let capture = || unsafe{ *ptr };
   x = 2;
   println!("{}", capture());
   println!("{}", x);
  }

(I had to use unsafe otherwise the borrow checker will complain will I modify x from underneath the closure; maybe a more elegant way to make the same point -- I don't really know rust...)

Actually hmm, I may want to take back my earlier comment. There are multiple things at play. There's scoping (where rust will copy across scope boundaries for non-ref types, which allows closing over something as in your first example above).

Then there's mutable refs and mutable variables, which as hope-striker mentioned I was confusing, possibly because I was using ints in my example. If instead we used a vec:

    fn main() {
     let x = vec![0,1,2]
     x.push(3) // fails since x isn't mutable
    }

There's no clear direct related concept here by default. If we're allowed to use pytype, you get this:

    def main():
      x: Sequence[int] = [1,2,3]  # Sequences aren't mutable
      x.push(3)  # fails since x isn't mutable

Cool, so mutable and immutable values are possible in both langs. What about refs? Well we went through that one, if you pass a mutable ref to a function in rust, you can modify the ref in ways that just aren't possible in python:

    fn main() {
        println!("Hello, world!");
        let mut x: i32 = 3;
        modifies(&mut x);
        println!("{}", x);
    }

    fn modifies(x: &mut i32) {
        *x = 5;
    }

There's nothing analogous to this in python. Everything is always passed as a mutable "value"[1], nothing is passed as a ref.

Cool so that's mutable variables and mutable references. That leaves this weird scoping issue. In rust (and in cpp) there's lots of scopes. Any set of braces creates a new scope, and so shadowing can happen across scopes. Lambda capture/closure happens over the scope. A given scope binds a name to a value, or a set of names to their values.

Python's a bit different, only new names are created in the scope. If a name isn't accessible in the given scope, the name is pulled from parent scopes etc.

So for the capturing behavior you want, there's weird nonlocal stuff that needs to be done, or you can explicitly make an additional scope, which removes the wonky behavior. If the name were really mutable, you'd be able to change what x referred to in the enclosing scope, which you can't.

tl;dr: This isn't mutable names, its python's (admittedly abnormal) scoping rules.

[1]: Unless you add in mypy or whatnot, where the typechecker will prevent you from modifying something that is non-mutable, but unlike in rust this isn't done with mutability as a first class citizen, its just that some interfaces expose mutating methods (`append`) and some don't. You can pass a list to a function that expects a list or a sequence, and the first case is mutable, while the second isn't.

Python's scope & mutability rules are idiosyncratic, but that's a distraction from what's going on here.

Let's go back to steveklabnik's ancestor comment:

"That's not an identical translation, the identical Rust would be:"

    fn main() {
       let mut x = 1;
       x = "foo";
    }

He was saying the identical Rust would not be:

    fn main() {
       let x = 1;
       let x = "foo";
    }

These are being compared to the following Python:

    x = 1
    x = "foo"

So consider these slightly enhanced versions of the fundamental question posed above.

Python:

  def mystery_py():
    x = 1
    capture = lambda: x
    x = 2
    return x * capture()

Rust:

  fn mystery_a() -> i32 {
     let x = 1;
     let ptr = &x as *const i32;
     let capture = || unsafe{ *ptr };
     let x = 2;
     return x * capture();
  }
  
  fn mystery_b() -> i32 {
    let mut x = 1;
    let ptr = &x as *const i32;
    let capture = || unsafe{ *ptr };
    x = 2;
    return x * capture();
  }

If you compare return values, you will find that mystery_py() returns the same as mystery_b().

So! I think you must agree that steveklabnik was right -- the rust code that is equivalent to the python code is the "let mut" variant. (Because surely you would not argue code that returns a different value is equivalent?!)

So now the question is, why?

Rather than answer, I will trollishly pose 2.5 more questions:

What would an implementation of mystery_a and mystery_b look like in scheme?

Would it be possible to author mystery_b in scheme if your "!" key was broken? (How about in some other purely functional language?)

I disagree that those are doing the same thing. I propose that the actual answer is c:

    use std::cell::RefCell;

    fn mystery_c() -> i32 {
        let x = RefCell::new(1);
        let capture = || x.borrow();
        x.replace(2);
        return *x.borrow() * *capture();
      }
  
    fn main() {
        println!("{}", mystery_c())
    }

Which is what I meant when I said that Box<T> might be the analogous thing (I guess it's actually RefCell, whoops!). And note that in this case, x is immutable :P

That said I accept your broader point, the effect is that python names act like mutable rust names, although the reality is slightly more complex (my final example is, I believe, the closest to actual reality).

You lost me, boss! Why do you think mystery_c is closer to mystery_py than mystery_b?

Let's go on a journey.

The answer is that I started with a hunch. You're treating x as a pointer sometimes, and a value other times. That seems strange, and unlike the python. In python the thing is always access the same way, it isn't a ptr type sometimes and a value type others.

So first let's talk about scopes. In python, you aren't introducing a closure. If we do introduce a closure, like with an IIFE:

    def mystery_closure():
        x = 1
        closure = (lambda v: lambda: v)(x)
        x = 2
        return x * closure()

suddenly we get 2. The IIFE/outer closure here is equivalent to the capture happening in rust. So this is more equivalent to the rust examples than your python example. Closures are what matter, not variable mutability.

Cool, so now let's add another wrinkle: `i32` in rust isn't a mutable type, there are no mutating methods on an i32. What happens if we use a type that has mutating methods, like a vec?

Let's start in python, since python doesn't allow multiline lambdas, we have to swap to using an inner function, which is fine, this makes the structure a bit clearer in python.

    def mystery_ mutable():
      x = [1]
      def closure():
        def inner(v):
          v.append(2)
          return v
        return inner(x)
      x.append(3)
      return x + closure()

And what if we do the same in rust? Well, we have to mark x as a mutable ref:

    fn mystery_b() -> Vec<i32> {
        let mut x = vec![1];
        let ptr = &mut x as *mut Vec<i32>;
        let capture = || unsafe{ (*ptr).push(2); 
                                  ptr };
        x.push(3);
        unsafe { x.extend(capture().as_ref().unwrap().iter()); }
        return x
      }

So the python value is a mutable ref, right? Well no, we're back to the whole issue of the closure being able to modify things outside itself in rust with a mut ref that we can't do with python:

    def mystery_mutable():
      x = [1]
      def closure():
        def inner(v):
          v = [5]
          v.append(2)
          return v
        return inner(x)
      x.append(3)
      return x + closure()

This returns [1,3,5,2] in python. If you translate it to rust with a mutable ref pattern, you'll get [5,2,5,2] and the 3 will just disappear:

    fn mystery_mutable() -> Vec<i32> {
        let mut x = vec![1];
        let ptr = &mut x as *mut Vec<i32>;
        let capture = || unsafe{ (*ptr) = vec![5,2];
                                  ptr };
        x.push(3);
        unsafe { x.extend(capture().as_ref().unwrap().iter()); }
        return x
      }

So in python, the thing isn't a const ref, but it's not a mutable ref, either, and it's certainly not a value type.

In languages like rust and cpp we describe calls as pass by reference or pass by value. Pass by value is mostly irrelevant here. When passing by reference, you can use a mutable or immutable reference. Immutable references don't allow you to modify the object, just read it. Mutable references allow you to modify or replace the object. With normal pointers and references, if you're able to modify the referenced object you can also replace it with an entirely new object.

The reasons for this are tricky, but have to do with self references in methods (self/this has to be mutable for a mutable method to work). In rust and cpp the self reference is exposed, so you can make it point elsewhere. In python you can't do this. This means that its tricky to pass an immutable reference to a mutable object in rust/cpp, but in python this is the only way things get passed around.

Rust calls this "interior mutability", and RefCell is the way to do interior mutability with references, as opposed to copyable types. The docs for RefCell actually call out passing &self to a method that requires mutability[1] as a use for RefCell, so in general you could use the RefCell to implement a python-like set of containers that can be passed "immutably" and still modified internally. In Pseudo-rust:

    struct PyVec {
      backing_arr: RefCell<Vec<T>>
    }

    impl PyVec {
      fn push(&self, v: T) {  // This isn't mutable?!
        backing_arr.borrow_mut().push(v);
      }
      ...
    }

Which would match python's semantics very closely

[1]: https://doc.rust-lang.org/beta/std/cell/index.html#implement...

Quality content! You should write this up into a blog post or something.

> The IIFE/outer closure here is equivalent to the capture happening in rust. So this is more equivalent to the rust examples than your python example.

Wait, I don't follow. Rewriting your example to only use one lambda for clarity, we have:

    def mystery_closure_one_lambda():
        x = 1
        def capture(v):
            return lambda: v
        closure = capture(x)
        x = 2
        return x * closure()
        

So notice the lambda (i.e. what we are assigning to the variable 'closure') is now capturing v, not x, which is why it doesn't see the change we make to x, i.e., why it returns 2 instead of 4.

But this is not equivalent to the rust code! There is no v at all in rust. We are capturing x! (It's slightly obscured by the fact that we to use an unsafe ptr to defeat the borrow checker, but we are still capturing x.)

So I do not think mystery_closure is equivalent to either of the rust mystery_a or mystery_b above; it is in fact equivalent to this:

  fn mystery_closure() -> i32 {
      let mut x = 1;
      let closure = (|v| move || v)(x);
      x = 2;
      x * closure() 
  }

Which also returns 2, just like the python code. (It's also a direct translation of the python code!)

> Let's start in python, since python doesn't allow multiline lambdas, we have to swap to using an inner function, which is fine, this makes the structure a bit clearer in python.

Careful! -- your de-lamba-fication accidentally changed the semantics. If we just de-lambda-fy, we get:

  def mystery_int():
      x = 1
      def closure():
          def inner(v):
              return v
          return inner(x)
      x = 2
      return x + closure()

Which returns 4, showing it's defnly not equivalent. The correct de-lambda-ficiation is:

    def mystery_closure_no_lambas():
        x = 1
        def capture(v):
            def inner():
                return v
            return inner
        closure = capture(x)
        x = 2
        return x * closure()

(which as a sanity check, returns 2, as it should).

Bringing in mutable reference data types like vec is I think not really relevent to what's at play here.

In both rust and python, the non-reference types mut i32 (rust) and int (python) are mutable. In rust you can pass a mutable reference to an i32, and in python you can't, but so what; that's not really relevent.

DIGRESSION:

Just for funsies, you actually can achieve what are essentially mutable references in python3 (you could also do this in py2 if you wanted to get nasty with locals()):

  # a mutable reference to a local variable
  class Ref:
      def __init__(self, getfn, setfn):
          self.getfn, self.setfn = getfn, setfn
      def get(self): return self.getfn()
      def set(self, v): self.setfn(v)
      value = property(get, set, None, "reference to local variable")

  # change a local variable using the mutable refernce
  def mutate(ref, new_value):
      ref.value = new_value

  def mystery_py_mutable_ref():
      x = 1

      # get a mutable reference 'ref' to x
      def get(): 
          return x
      def set(v):
          nonlocal x
          x = v
      ref = Ref(get, set)

      # capture x in a closure
      capture = lambda: x

      # mutate x
      mutate(ref, 2)

      # finally evaluate x and the closure; this will return 4!
      return x * capture()

END DIGRESSION

But anyway, I don't think it's actually relevent here.

Question: Are you familiar with scheme? Would you agree or disagree that the following scm_mystery_a and scm_mystery_b are equivalent to the rust mystery_a and mystery_b functions?

  (define scm_mystery_a 
      (lambda ()
          (let ((x 1))
          (let ((capture (lambda () x)))
          (let ((x 2))
          (* x (capture)))))))

  (define scm_mystery_b
      (lambda ()
          (let ((x 1))
          (let ((capture (lambda () x)))
          (set! x 2)
          (* x (capture))))))

  (display (scm_mystery_a)) (newline)
  (display (scm_mystery_b)) (newline)

> So notice the lambda (i.e. what we are assigning to the variable 'closure') is now capturing v, not x, which is why it doesn't see the change we make to x, i.e., why it returns 2 instead of 4.

Yes, but this goes back to the scoping issue: in python, lambdas (and functions in general) don't capture. The only way to close over something is to pass as an argument. So to get the lexical closure behavior that rust provides, you have to add extra stuff in the python. This indeed makes the translations not mechanical (and you can add the lambda back in the rust, it doesn't hurt anything in these examples), but to get matching scoping behavior between rust and python, you need an extra layer of indirection in the python.

> Bringing in mutable reference data types like vec is I think not really relevent to what's at play here.

Of course it is, because in python everything is a reference. There's no such thing as a value type, and this is precisely where the difference in behavior comes in (other than the scoping issues). A rust RefCell is the thing that most naturally matches the actual in memory representation of a PyObject.

As for your digression, eww, although you forgot to actually do the sneaky part. This would be the actual demonstration, you need to modify the list in the closure (a real closure), and set it after the closure is created and before it is evaluated:

      # capture x in a closure
      def closure(v):
        def inner():
          mutate(v, 2)
          return v
        return inner

      capture = closure(ref)

      ref.set([3])

> Are you familiar with scheme?

Unfortunately not.

Run my examples, they work! Python absolutely has real closures...

Yes your digression example works, but it works equally well without a mutable ref. You need to mutate the thing in the callback (and to have the callback actually close over the ref) to require mutable ref semantics.

And yes, python closures don't capture environemtnal vars like rusts do. If you need them to capture env vars, you have to do what I did in the example.

I think our understandings are too far apart to converge in a HN thread. Nevertheless, it has been a pleasure!