What is difference between datum->syntax and syntax #' in define-syntax body?

Question

Testing code:

(define-syntax (test-d stx)
  #'(begin 
      (define (callme a)
        (writeln a))))

(define-syntax (test-e stx)
  (datum->syntax stx '(begin 
                         (define (callme2 a)
                           (writeln a)))))


> (test-d)
> (callme 1)
. . callme: undefined;
 cannot reference an identifier before its definition
> (test-e)
> (callme2 1)
1

I do not understand difference in test-d and test-e. To me they look equal. Still, callme is not defined.

Even macro stepper says it is the same.

Expansion finished
(module anonymous-module racket
  (#%module-begin
   (define-syntax (test-d stx)
     #'(begin (define (callme a) (writeln a))))
   (define-syntax (test-e stx)
     (datum->syntax
      stx
      '(begin (define (callme2 a) (writeln a)))))
   (begin (define (callme a) (writeln a)))
   (begin (define (callme2 a) (writeln a)))))

I guess that in test-d is missing some information (context?) that is passed in test-e through stx.

How can I achive to have callme defined also with using #' only?

Answer 1

Racket’s macro system is hygienic. This means that identifiers introduced by a macro live in their own scope—they do not collide with identifiers used or defined outside the macro. This is usually what you want, since it avoids problems when both a macro author and a macro user decide to use the same variable names.

In your case, however, you want behavior that is explicitly unhygienic. You want the macro to define a fresh identifier and have that identifier be in scope outside of the macro. Fortunately, while Racket enforces hygiene by default, it allows you to break (or “bend”) hygiene when you want to.

When you use #', aka syntax, you are using hygienic macro features. This means that your definition of callme is only visible inside of test-d, and it won’t be visible to the calling code. However, datum->syntax is one of the primary mechanisms that allows you to break hygiene: it “forges” a new piece of syntax that lives in the same scope as another piece of syntax, in your case stx, which is the input to the macro. This is why callme2 is visible outside of test-e’s definition.

However, this is a heavy hammer… too heavy, in fact. Your test-e macro is brutally unhygienic, and this means it can be broken if the user of the macro binds a name used by test-e. For example, if the user defines a local variable named begin, test-e won’t work anymore:

(define-syntax (test-e stx)
  (datum->syntax stx '(begin 
                        (define (callme2 a)
                          (writeln a)))))

(let ([begin 42])
  (test-e)
  (callme2 1))

define: not allowed in an expression context

You can avoid this problem by being more conservative about how you break hygiene. Really, in this situation, the only piece of the macro we want to be unhygienic is the callme2 identifier, so we can forge that piece of syntax using datum->syntax, but use #' for all the rest:

(define-syntax (test-e stx)
  (with-syntax ([callme-id (datum->syntax stx 'callme2)])
    #'(begin
        (define (callme-id a)
          (writeln a)))))

(let ([begin 42])
  (test-e)
  (callme2 1))

Now the program works, and it is only unhygienic in the one spot that it needs to be.