arXiv:2106.11053v3 [cs.LG] 3 May 2022

Leveraging Language to Learn Program Abstractions and Search Heuristics

Catherine Wong 1 Kevin Ellis 2 Joshua B. Tenenbaum 1 3 Jacob Andreas 1

AbstractInductive program synthesis, or inferring pro-grams from examples of desired behavior, offersa general paradigm for building interpretable, ro-bust, and generalizable machine learning systems.Effective program synthesis depends on two keyingredients: a strong library of functions fromwhich to build programs, and an efficient searchstrategy for finding programs that solve a giventask. We introduce LAPS (Language for Abstrac-tion and Program Search), a technique for usingnatural language annotations to guide joint learn-ing of libraries and neurally-guided search modelsfor synthesis. When integrated into a state-of-the-art library learning system (DreamCoder), LAPSproduces higher-quality libraries and improvessearch efficiency and generalization on three do-mains – string editing, image composition, andabstract reasoning about scenes – even when nonatural language hints are available at test time.

1. IntroductionMachine learning approaches based on program synthesis–the automatic inference of symbolic programs–can offerrobustness, interpretability, verifiability, and strong gener-alization in few-shot learning settings (Appel et al., 2017;Lake et al., 2017). Many machine learning tasks can beformulated as program synthesis problems, including datamanipulation (Delaware et al., 2015; Gulwani et al., 2017),semantic parsing (Artzi & Zettlemoyer, 2013; Liang, 2016),structured visual understanding (Johnson et al., 2017b; Yiet al., 2018), image generation (Ellis et al., 2017; Ganinet al., 2018), and policy learning (Fikes & Nilsson, 1971;Cropper & Muggleton, 2015; Silver et al., 2020).

This paper introduces Language for Abstraction and Pro-gram Search (LAPS), a framework for improving the ef-ficiency and generalizability of learned program synthesis

1MIT 2Cornell University 3Center for Brains, Minds andMachines (CBMM) - MIT. Correspondence to: Catherine Wong<[email protected]>.

Proceedings of the 38 th International Conference on MachineLearning, PMLR 139, 2021. Copyright 2021 by the author(s).

models using natural language supervision. In LAPS, lan-guage guides learning of both libraries of reusable programabstractions and heuristics for searching in the space ofprograms. High-quality program libraries and search meth-ods are the main ingredients of effective program synthesisapproaches (Gulwani et al., 2017). Recent approaches toprogram synthesis have attempted to learn search models(Gulwani et al., 2015; Polozov & Gulwani, 2015; Baloget al., 2016; Devlin et al., 2017), program libraries, or bothjointly from data (Shin et al., 2019; Dumancic & Cropper;Ellis et al., 2021; 2020; Lazaro-Gredilla et al., 2019), buteven the current best learning approaches can be computa-tionally inefficient (often requiring upwards of thousands ofCPU hours to bootstrap learning) and do not always discovergeneralizable libraries or search strategies.

LAPS builds on the intuition that natural language offers apowerful source of information for tackling both learningproblems. Language simultaneously provides an efficientchannel for communicating the structure of the search space(an instruction like draw a large hexagon next to a smallpentagon decomposes a complex graphics task into high-level parts) and a lexicon that names important reusableconcepts in a given domain (for instance, suggesting thata function to draw variable-sized polygons might be use-ful for future graphics tasks). In this work we show howinducing jointly compositional generative models over natu-ral language and programs provides a strong scaffold forlibrary and search model learning in a hierarchical programinduction model. When integrated into a state-of-the-artlearning algorithm, DreamCoder (Ellis et al., 2021; 2018),our approach dramatically improves performance on threedifferent synthesis domains: string editing, structured imagegeneration and scene understanding. Compared to the basesynthesis approach, LAPS solves and learns more quicklyfrom synthesis tasks, and produces higher-quality librariesthat improve generalization to downstream tasks withoutnatural language hints.

LAPS builds on several recent developments in (non-language-based) program synthesis, so we begin with areview of related work (Sec. 2), then formalize the searchand library learning problems (Sec. 3) and base synthesisalgorithm (Sec. 4). We then describe how LAPS extends thebase algorithm to include language in learning (Sec. 5) andconclude with empirical results (Sec. 6).

arX

iv:2

106.

1105

3v3

[cs

.LG

] 3

May

202

2


+ Iteratively learned jointly compositional generative models over program library and language

(ii) Abstraction is structured over language to learn functions that compose like language

(i) Neural search learns from generated language-annotated programs to condition on language as a high-level training signal

Language-annotatedtraining tasks

+ Iteratively learned library as a generative prior over programs

(ii) Abstractions learned from training programs may be overfit to training tasks

(i) Conditional neural search learned from program samples can struggle to generalize to hard training tasks

Training tasks with no ground truth programs

A. Base learned synthesis algorithm (DreamCoder)

a small nine gon next to a small square

four nested squares

a five sided snowflake with a short line and a small seven gon as arms

move_pen(for ∞ (move_pen (* unit_line 3) (/ 2π 6)))

learned_fn_0 = (for ∞ (move_pen (* unit_line 3) (/ 2π x )))

gon_fn = (for ∞ (move_penx (/ 2π y )))

large six gon

B. Language for abstraction and program search (LAPS)

(for ∞ (move_pen (* unit_line 3) (/ 2π 6)))

large_fn = (* unit_line 3)

....leverages compositional generativity of programs to learn

....leverages compositional generativity of language to learn programs

L Library

Program

Executed example

Joint library-language model

(Program,language)

Executed example and annotation

J

large six gon

forward sampleprograms

learned execution-conditioned inverse

large six gon (for ∞ (move (* unit_line 3) (/ 2π 6)))forward sample

programs and language

“large six gon”

learned execution and languageconditioned inverse

(for ∞ (move _pen(* unit_line 3) (/ 2π 6)))

unit_line

for

*

add back to learned library

add back to learned library

learned_fn_0

gon_fn

“large”“line”

large_fn

abstract overdiscoveredprograms

abstract jointlyover programs and language

“gon”move_pen

unit_line

for

*

Figure 1. Our model, Language for Abstraction and Program Search (LAPS) integrates natural language into base learned synthesisalgorithms formulated as hierarchical Bayesian inference (A, left) for jointly learning a library of program abstractions and a neuralsearch heuristic for synthesis. We give an extended formulation (B, left) defined jointly over the program library and natural languagedescriptions of synthesis tasks, that can be used to incorporate natural language into both abstraction and search heuristic learning. Whenincorporated into a concrete learning algorithm, DreamCoder (A, right) we show that LAPS allows the model to leverage language richlyduring training to improve the generalization of both the learned neural search model and the learned library of program abstractions.

2. Related WorkOur work draws on recent program synthesis approachesthat learn to synthesize programs from examples using neu-ral models to guide search (Gulwani et al., 2015; Baloget al., 2016; Parisotto et al., 2016; Devlin et al., 2017; Polo-sukhin & Skidanov, 2018; Abolafia et al., 2018; Nye et al.,2019; Ellis et al., 2019; Si et al., 2019; Ye et al., 2020a); andlearn libraries of symbolic abstractions from a collectionof related programs or tasks (Dechter et al., 2013; Zhanget al., 2017; Shin et al., 2019; Dumancic & Cropper; Elliset al., 2018; 2021). Our formulation builds on hierarchi-cal Bayesian formulations of program learning that frameboth synthesis and library learning as probabilistic inference(Liang et al., 2010; Lake et al., 2015; Ellis et al., 2021).

Natural language has also been used to scaffold latent repre-sentation learning (Frome et al., 2013; Jia & Liang, 2016;Andreas et al., 2017; Ye et al., 2020b; Goyal et al., 2020;Liang et al., 2020; Mu et al., 2019; Luketina et al., 2019),and as a high-level specification for program synthesis tasks(Ye et al., 2020a; Nye et al., 2019; Polosukhin & Skidanov,2018; Ye et al., 2020b; Desai et al., 2016; Srivastava et al.,2017). Here we present an approach that integrates languageannotations in training for learning a more generalizablelibrary and program search model that can be used aftertraining with no additional annotations for new tasks.

3. Inductive synthesis and library learningConsider the problem of writing a graphics program to drawthe large hexagon image in the left column of Fig. 1. Thisis an inductive program synthesis problem: a task t (likedraw a large hexagon) is specified with examples of whata program should do, where each example is given as aninput x (in this case, the blank image canvas) and the desiredoutput y (the large hexagon image). A program ρ solvesthe task if it produces outputs that are consistent with thespecification when executed – that is, if evaluating ρ underan execution model E yields JρKE(x) = y.

Program synthesis begins with a library L = {l0, ..ln}containing the set of primitives that can be combined to pro-duce solution programs, such as the (pseudo-code) primitivefunctions in a simple graphics language:

L = move pen|unit line|for|*|π|∞|0|1|2|...

which draw lines on a canvas parameterized by their lengthand angle. Given a library, there is also the problem ofsearch: effective program synthesis requires a search strat-egy S that can be given a task specification (such as theimage of a hexagon) and automatically discover a solutionprogram like the one shown in Fig. 1:

(for ∞(move pen(∗ unit line 3)(/ 2π 6))

by searching over programs built from functions in L.


Both of these ingredients – the library L, and the searchstrategy S – can be made much more efficient if the syn-thesis engine will be expected to solve multiple relatedproblems. In the graphics domain, for example, synthesis ofthe various images depicted in Fig. 1 is much more easilyaccomplished using a library like

L = polygon|large line|small line...

in which the original hexagon task can be expressed as

polygon(6, large line)

A good library already provides a foundation for efficientsearch by making solutions easier to express. Even withsuch a library, search can be further guided by informationabout the prior structure of programs (for example, the factthat polygon is typically called with a large line orsmall line function as a second argument) and by infor-mation about the target task itself (for example, the fact thatthe target image contains six line segments). Thus, one wayto describe an effective search strategy S is via a prior overprograms P[ρ|L] in the library and a conditional inferencemodel for inferring P[ρ|t,L], the distribution over programslikely intended by the observed task examples t.

The foregoing discussion lays out the basic ingredients ofa hierarchical Bayesian formulation of program synthesis(used in learning algorithms like (Ellis et al., 2021; Lakeet al., 2015; Dechter et al., 2013); see the graphical modelin Fig. 1A, left) for jointly learning a library and conditionalsearch model from a dataset T of synthesis tasks. We denotea prior over programs as P[ρ|L, θL], on a library L withparameters θL. Given the observed tasks, we define thelikelihood of the latent library and parameters as:

Φ(L, θL) = P[L, θL]∏

t∈T

∑

ρ

P[t|ρ]P[ρ|L, θL] (1)

where P[L, θL] is a prior over all possible libraries and pa-rameterizations, and P[t|ρ] is the likelihood that each induc-tive task t is consistent with a program ρ (for our purposes,P[t|ρ] = 1 if the program produces the desired output ex-amples and 0 otherwise.) Learning in this model meansestimating the optimal library and its parameters

L∗, θL∗ = arg maxL,θL

Φ(L, θL) (2)

along with a conditional model P[ρ|t,L∗] that can inferprograms for new tasks.

This formulation also foreshadows a straightforward wayin which linguistic descriptions of tasks (like those in thefirst column of Fig. 1) could be integrated into learning: wecould simply extend the conditional model as P[ρ|t, dt,L∗]to include a task’s description dt. We come back to this (and

describe a more complete integration) in our approach, butfirst describe a concrete implementation of Eq. 2 on whichwe can realize the language-enriched model.

4. Base learning algorithm: DreamCoderThe LAPS framework we describe in this paper is a generalone for extending Bayesian models of program learninglike the one in Eq. 2 to incorporate information from lan-guage. For concreteness, however, our presentation andexperiments build on the specific DreamCoder algorithm ofEllis et al. (2021), which we briefly review here. We chooseDreamCoder because it exposes a modular implementationof the library and search learning problems in Eq. 2 and haspreviously demonstrated state-of-the-art performance acrossa variety of synthesis domains (Ellis et al., 2021; 2020).

DreamCoder is initialized with a base library L0 of startingprimitives and a dataset of training tasks T . It returns alearned final library Lf augmented with program abstrac-tions and a learned neural search model Q(ρ|t,L) that pre-dicts high probability programs conditioned on the taskexamples. Learning is iterative: DreamCoder alternatelysearches for solution programs to the training tasks (givena current library Li and search model Qi) and updates thelibrary and search model based on new solved tasks. Wegive details on each component below.

4.1. Program prior

DreamCoder defines the prior over programs as a proba-bilistic context free grammar (PFCG; Johnson 1998) forprograms generated as productions from a library L of func-tions l ∈ L 1. Formally, DreamCoder assigns a real-valuedweight θLi to each library function, which when normal-ized yields a production probability P[l|L, θL]. The priorprobability of a program ρ is given by

P[ρ|L, θL] =∏

l∈ρP[l|L, θL] (3)

the weighted product of probabilities of all of its constituentlibrary functions. As all P[l|L, θL] < 1, this is equivalentto a description length prior over programs: longer pro-grams (with more constitutent elements) will have lowerprior probability under Eq. 3 since P[l|L, θL] monotonicallydecreases as |ρ| = |{l ∈ ρ}| increases.

4.2. Amortized conditional inference

To identify programs that solve tasks t while obtaining highprobability under P[ρ|L, θL], DreamCoder trains a neural

1In addition to initial and learned functions, Ellis et al. (2021)define L to also include any initial literals and a rule for generatingvariables, such that programs can be completely generated asproductions from the PCFG. We use the same formulation.


search heuristic Qi(ρ|t,Li) at each iteration i to approx-imate the inverse conditional model. The heuristic usesa neural model trained to predict programs written in thecurrent library Li according to the posterior:

Qi(ρ|t,Li) ≈ P[ρ|t, (Li, θLi)] ∝ P[t|ρ]P[ρ|(Li, θLi)](4)

conditioned on an encoding of the training examples (e.g.an embedding of the image in the task specification). Thismodel is trained in the distant supervision setting (whichbegins with no supervised program data) by leveraging theforward generative model: sampling programs from theprior, executing them to produce observed tasks, and thenminimizing Q(ρ|t,L) in Eq. 4 on the sampled programs,conditioned on their executions. This generative trainingprocedure is generally applicable to any neural implemen-tation of Q(ρ|t,L). (But see Ellis et al. (2021) and oursupplementary material for additional details on the modelarchitecture, which we reimplement in our experiments.)

4.3. Abstraction learning as program compression(maximizing the likelihood of programs)

The DreamCoder algorithm also iteratively updates the li-brary (Li, θLi

) to approximately optimize Eq. 2 (findingL∗, θ∗L which maximize the likelihood of the inferred latentprograms). Ellis et al. (2021) leverage equivalence to a com-pression problem defined over programs and the library. Asdiscussed in 4.1, the PCFG program prior is equivalent to adescription length prior over programs. Ellis et al. (2021)place an additional Dirichlet prior over the library descrip-tion length:

P [L] ∝ exp

−λ

∑

ρ∈Lsize(ρ)

(5)

Estimating the optimal library then becomes the problemof inferring new library abstractions which can jointly com-press the latent training programs (rewritten under the newlibrary Li+1) and the description length |Li+1| of the up-dated library (to optimize for shared abstractions acrossprograms). This objective would still require inference overall possible ways of refactoring the latent programs underthe updated library. Ellis et al. (2021) approximate this byonly considering candidate abstractions and program refac-torings that can be found via an efficient lambda-abstractionalgorithm. As an example, this could refactor the largehexagon program

(for ∞(move pen(∗ unit line 3)(/ 2π 6))

to expose a candidate abstraction like

λx.(for ∞(move pen(∗ unit line 3)(/ 2π x))

while also rewriting the original program using this abstrac-tion. Notably, this fragment – which draws polygons with

lines of length 3 for sides – is not the most intuitively gener-alizable for the graphics domain. A programmer with moredomain-specific prior knowledge would probably prefer anabstraction like

λxy.(for ∞(move pen(∗ unit line y)(/ 2π x))

which additionally parameterizes the polygon by the lengthof its sides, and is semantically equivalent to the high-levelpolygon fn described in the problem setup in Sec. 3.However, learning abstractions by compressing the libraryand current solved training tasks may actually disfavor thismore intuitively generalizable (but less compressive) candi-date. Our second key goal in introducing language will beto leverage it as an additional source of prior knowledge toimprove abstraction generalization.

5. Our Approach: Language for Abstractionand Program Search

Our work considers how the general learning problem –jointly learning the library L which defines the prior overprograms and the conditional search strategy S which in-verts from tasks to programs – can be enriched in thelanguage-annotated setting. Here, at least a subset of thetraining tasks are additionally annotated with a natural lan-guage description dt (such as the natural language descrip-tion large six gon for the large hexagon drawing task in Fig.1B). Language offers a more direct source of informationfor discovering a library like the one in our setup,

L = polygon|large line|small line...

if we leverage the expectation that generalizable abstractions(like a candidate polygon function) should correspondsystematically to named fragments in natural language (likethe token gon).

Language can also be leveraged by the conditional searchmodel: learning systematic correspondences between lan-guage and programs from descriptions like large six gonshould inform search on new tasks (like the one describedas a small nine gon next to a small square in Fig. 1B) on thebasis of shared language (like gon).

Our approach, LAPS (Language for Abstraction and Pro-gram Search) formalizes these intuitions by extending thehierarchical Bayesian problem formulation over programsgiven in Sec. 3 to additionally generate natural languagetask descriptions (see graphical model in Fig 1B, left). Inparticular, we assume the existence of a jointly generativemodel J(ρ, dt) over latent programs that solve tasks, andcorresponding natural language descriptions. We rewrite theoriginal prior over programs P[ρ|L, θL] defined on a libraryL to a joint prior P[ρ, dt|J, θJ ], and extend the distributionin Eq. 1 over the latent joint model J with parameters θJ ,


written as

Φ(J, θJ) = P[J, θJ ]∏

t∈T

∑

ρ

P[t|ρ]P[ρ, dt|J, θJ ] (6)

Learning in the language-augmented setting now involvesestimating the optimal joint model and its parameters

J∗, θJ∗ = arg maxJ,θJ

Φ(J, θJ) (7)

along with a language-conditioned model P[ρ|t, d, J∗] thatcan infer programs for new tasks based on both specificationexamples and task descriptions.

In the remainder of this section we first describe a generaljoint model formulation that can be learned from language-annotated training tasks. We then show how the joint frame-work allows natural language to inform learning at both theabstraction and search level in a concrete example, usingDreamCoder as the base hierarchical algorithm.

5.1. Joint prior over programs and language

Base prior We formulate our joint prior over languageand programs as

P[ρ, dt] = P[ρ|L, θL]P[dt|ρ,L] (8)

decomposed as the product of the original programprior defined on a program library P[ρ|L, θL], and alearned program-to-natural-language “translation” modelT (dt|ρ,L) ≈ P[dt|ρ,L] which describes how natural lan-guage descriptions are generated for latent programs (inour running example, this model would describe how thelarge six gon description was generated conditioned on theprogram solution for that task.) This decomposition buildsmodularly on the original program prior defined only on thelibrary L. Learning T (dt|ρ,L) formalizes the intuition thatthere should be a learnable relationship between languagethat describes tasks and latent programs that solve them.

T (dt|ρ,L) can be implemented in many ways (e.g. (Wong& Mooney, 2007; Joshi & Schabes, 1997; Bahdanau et al.,2014; Chen et al., 2018)), compatible with the vast literatureon structured translation between languages, including natu-ral languages and programming languages. Our experimentsuse the translation model popularly known as IBM Model 4(Brown et al., 1993), one of a class of well-studied Bayesianmachine translation models (Gal & Blunsom, 2013) whichdecompose T (dt|ρ,L) into

T (dt|ρ,L) ∝∏

w∈dt,l∈ρPT [w|l] (9)

a product of learned token-level translation probabilitiesPT [w|l] between individual functions l in a task’s latent

program ρ and words w in the task description dt. (See sup-plementary materials for model implementation and train-ing details.) This token-level decomposition more directlycaptures the intuition in our setup: that abstractions in aprogramming library generally correspond systematicallyto individual names in natural language descriptions, andthat the inverse conditional search can be guided based ona generally compositional relationship between programprimitives and words. This formulation also allows thesecompositional relationships to be inferred from fewer ob-served examples than would be possible with other transla-tion models with weaker inductive biases. However, Eq. 8should extend to include any similar translation model andneed not include this stronger decomposition.

Adding richer priors In LAPS, the joint model can alsoprovide a controllable interface for incorporating additionalprior knowledge about language into learning. Learnedtranslation models are often fit to only maximize the likeli-hood of the observed language (here, with respect to inferredlatent training programs). However, our formulation alsosupports T (dt|ρ,L) enriched to include additional priorsover language (such as speaker-specific language usage, orpragmatics models that capture a speakers’ other commu-nicative goals (Grice, 1989; Goodman & Frank, 2016).)

In our experiments (Sec. 6.1) we showcase this with resultsfrom an extended model incorporating an additional mutualexclusivity prior. Mutual exclusivity models the expectationthat newly encountered words should correspond to differentmeanings than known ones. This prior has been shown toplay an important role in language learning in cognitivescience (Frank et al., 2009; Markman & Wachtel, 1988),and in machine learning models (Gandhi & Lake, 2019).

In the synthesis setting, mutual exclusivity can capture theexpectation that “new” words (which appear in descriptionsof currently unsolved tasks) are more likely to correspondto different program components than those used in solvedtraining tasks (and for which there would otherwise be nosignal to learn a translation model in the distant setting).Our extended model incorporates this prior by updatingEq. 9 to distinguish between Wknown (words that appearin solved training tasks with latent programs) and Wnew

(newly encountered words) as

TME(dt|ρ,L) ∝∏

w∈d,l∈ρ(1[w ∈Wknown]PT [w|l])

(1[w ∈Wnew]P[l|L, θL]−1])

(10)

where new words are modeled as inversely related to primi-tives under the program prior (fit to previously solved tasks)– modeling the expectation that new words more likely relateto less-used program components than those used so far.


5.2. Integrating the joint model into amortizedconditional search

The joint model allows LAPS to incorporate natural lan-guage into the learned conditional search model over pro-grams. In place of the original neural amortized model in thebase algorithm (Sec. 4.2), we train an extended, language-conditioned modelQi(ρ|t, dt, Ji) at each iteration to predictprograms according to:

Q(ρ|t, dt, Ji) ≈ P[ρ|t, dt, J, θJ ]

∝ P[t|ρ]P[ρ, dt|J, θJ ]

∝ P[t|ρ]P[dt|ρ]P[ρ|L, θL]

≈ P[t|ρ]T (dt|ρ,L)P[ρ|L, θL]

(11)

which amortizes program inference under our joint modelformulation. Importantly, we can train this neural modelusing samples from the joint generative model, consisting ofsampled programs and corresponding generated language.As with the original learning setting, this sample-basedtraining allows LAPS to learn a generalizable, language-conditioned neural search heuristic, capable of leveragingcompositional patterns in natural language, from very fewexamples in the distant supervision setting. We can alsonow see the benefits of richer language-specific priors (suchas mutual exclusivity): the neural model trained to amortizeinference from the joint generative model can also approxi-mate the mutual exclusivity bias, enabling better explorationand generalization in the presence of new words.

5.3. Abstraction learning as joint model compression

The extended joint model objective in Eq. 2 and 7 also al-lows LAPS to incorporate natural language into abstractionlearning. Extending the compression-based abstraction ob-jective in the base algorithm – which optimized for librariesthat maximally compress the latent training programs and li-brary – requires defining a prior over the language-programtranslation model T in terms of the optimal program library.

We place a prior over T defined on a program library L anda natural language token vocabulary W as

P[T |L] ∝∑

l∈L,w∈W−I(PT [w|l]) (12)

where −I(PT [w|l]) = − log(PT [w|l]). This models theintuition that a good library contains program abstractionswhich correspond well to individual language tokens, and re-duce entropy in the compositional translation model. Defin-ing the prior compositionally also allows the algorithm tomaintain the desirably property from (Ellis et al., 2021), inwhich the joint likelihood can be efficiently re-approximatedwith respect to individual candidate program abstractionsbased on their constituent subcomponents l and correspond-ing translation distributions PT [w|l] under the current trans-lation model. As in the base synthesis algorithm, we

Algorithm 1Input: Initial library L0, annotated training tasks (T,D)Initialize J ← uniform; training task solutions p← {}for i ≤ f doQi(ρ|t, dt)← Train on (p, T, dt) and samples ∼ Jp← programs from search amortized with QiLi ← abstractions optimized over (p, J)p← programs rewritten using abstractions from LiJi ← Fit θL and T (dt|ρ) to (p, dt)

end forReturn Qf ,Lf

fully re-estimate a new translation model at each iterationTi+1(dt|ρi+1,Li+1) to fit the updated library and refactoredprograms. See the supplement for extended details.

Taken together, Alg. 1 summarizes the concrete algorithmusing LAPS to incorporate language into (Ellis et al., 2021).

6. ExperimentsWe demonstrate LAPS on three different domains: stringediting, compositional graphics drawing, and scene rea-soning, which we choose to represent a diverse range oftasks and accompanying language (Fig. 2). In all three do-mains, we find that compared to the base synthesizer, LAPSlearns and solves heldout synthesis problems faster (Table1, Sec. 1-2), and produces higher-quality libraries that im-prove generalization even when natural language hints arenot available after training (Table 1, Sec. 3).

Below we summarize each domain. We then discuss resultsshowing that LAPS is effective because of how the hier-archical model incorporates language during learning: wefind that (1) LAPS searches more effectively during training,enabling it to solve and learn from more diverse trainingtasks than the baseline model; (2) LAPS abstracts moreeffectively during training, adding in more generalizablelibrary routines as it learns; and (3) LAPS can use languageduring testing if it is available, as an important additionalsource of high-level information during synthesis.

6.1. Domains

All three domains consist of a dataset of inductive synthe-sis tasks t specified as input/output examples; procedurallygenerated synthetic language annotations; and human lan-guage annotations sourced from Mechanical Turk. We usesynthetic language as our primary evaluation benchmark:we are interested in a controlled probe of learning whenwords are systematically reused and composed, but refer tomore abstract concepts than in the initial base programminglanguage. However, we also use human language to evalu-ate the practicality of our approach in real-world settings.


A. String Editing (shown with sample I/O examples of n=30 and random human description of n=3)

pavings → pavinbforgiveness → forgivenebenterprises → enterprises

if the word ends with consonant s replace that with b

if the word ends with a consonant and s then change them both to b

cools → gcoolscultivator → gcultivatorbloomed → bloomed

(Synth) if the word starts with consonant vowel add g before that

(Human) if word begins with consonant followed by vowel , add an g to the beginning

topazes -> topazsuburbs -> suburbsreckless -> reckls

if there is e s remove that

remove the e s from the word

shouldering -> shoululderinghath -> hathoutrun -> oututrunun

if there is u any letter double that

the next letter with the letter u should be repeated as a pair for this transformation

C. Compositional Graphics (shown with random human description of n=3)

Simple shapes Complex objects Compositional objects and relations

a small trianglesmall triangle

a medium square one medium square

a medium eight gon octogon

a big circle just a circle

a seven pointed stara seven sided snowflake with long triangles as arms

a four stepped zigzagfour step ladder going from top to bottom

a greek spiral with eight turnsa long line that curls in on itself at right angles

a small five gon next to a small seven gona five sided gon beside a seven sided gon

a small nine gon separated by a big space from a small circlenine gon on left with small circle on right not connected

a small triangle connected by a big line to a medium trianglea small triangle with a long line and a medium triangle

six small five gons in a rowsix overlapped pentagons going left to right

seven sided snowflake with a short space and a short line and a short space and a small triangle as armsa seven sided snowflake with seven triangles and line

four nested squaresfour stacked squares

B. Scene Reasoning (shown with sample I/O examples of n=7 and random human description of n=2)

Original CLEVR (sample templates from full set) Extended scene manipulation and counterfactuals

What number of gray rubber cubes are there?

how many grey rubber cubes do you see

2

1

2

There is another thing that is the same color as the large rubber thing; what is it made of?what material is the other object that is the same color as the large rubber object

metal

rubber

metal

What if the gray sphere became a small green metal sphere?

what if the grey ball morphed into a small green ball

If you removed the red things, how many spheres would be left?

count the spheres would be left after removing the red things

3

3

0

a small semicircle(f19 (f9 0 x))

a medium semicircle(f3 (f9 0 x))

a big semicircle(f9 (* (/ ε 1) 5) x)

f0=(λ (x y z) (for x (λ (u v) (move z y v))))

1

. . .

for

pen-up

f4=(λ (x y z) (f0 x (/ 2π y) 1 z))

f5=(λ (x y) (f4 x x y))

0.27 | gon0.22 | small

f9=(f0 ∞ ε)

0.09 | small

0.07 | semicircle

f24=(λ (x y) (f23 (λ (z u) (f21 y 0 x u))))

f17=(λ (x) (pen-up (λ (y) (f16 x y))))

0.67 | separated0.06 | space

0.09 | snowflake0.09 | arms

D. Example initialgraphics primitives

shown with learned high probability p(word | primitive)

...

...

a small five gon(f5 5 x)

a small nine gon(f5 9 x)

a medium seven gon(f5 2 (f20 7 x))

eight sided snowflake with a small seven gon as arms(f24 7 8 x)

five sided snowflake with a short line and a medium five gon as arms(f24 5 (λ (x) (get/set (λ (y) (f2 1 (f41 5 y)))x)) z)

....and example program abstractions learned with language

rotates and draws a unit line

lift pen betweenconsecutive shapes

rotational symmetry by number of sides

move pen in parameterized loop

smooth curve rotate shapesaround axis

move

2

+

-

Figure 2. (A, B, C) Example tasks from all three synthesis domains shown with synthetic and sample human language annotations.Inductive synthesis domains are shown with a random subset (n=3) of the paired input/output examples. Human language annotationsare also randomly sampled (all domains were annotated by multiple people for a broader range of language.) (D) Representativeinitial program primitives and library abstractions learned with LAPS for the graphics domain. Shown with example tasks solved withsynthesized programs containing the learned abstractions and high probability natural language learned from the joint model.


Additional information for all domains is in the supplement.

String editing: structured string transformation problemstaken from (Andreas et al., 2017) (n=1000 train; n=500 test).Tasks consist of input dictionary strings transformed usingrandomly sampled regular expression transducer (30 I/Oexamples per task). We choose this domain to demonstrateLAPS on an important classic synthesis domain (Lau &Weld, 1998). The dataset of Andreas et al. (2017) containshuman annotations; synthetic language annotations are gen-erated over the ground-truth regexes using templates basedon the original human annotations. We initialize synthesiz-ers with functional programming primitives (map, fold, cons,car, cdr, length, index) and character constants (followingthe simpler text editing domain in the baseline paper (Elliset al., 2021)). The neural search model encodes the I/O taskexamples as character arrays with a bidirectional GRU.

Compositional graphics: inverse graphics problems(n=200 train; n=111 test) where each task is specified byan image and solved by synthesizing a program in LOGOTurtle graphics (Abelson & DiSessa, 1986). This is inspiredby the graphics domain in (Ellis et al., 2021) but re-designedto be more challenging (ground-truth programs are muchlonger on average in the base programming language) andexplicitly compositional. Synthetic language annotationsare generated with high-level templates over the objects andrelations in each task; human annotations are sourced asimage descriptions from MTurk. We initialize synthesiz-ers with the graphics primitives in (Ellis et al., 2021). Theneural model encodes image examples with a CNN.

Structured scene reasoning: inductive scene reasoningtasks (n= 212 train; n=115 test) where each synthesis prob-lem is specified by a structured input scene, and outputs canbe a number (how many red rubber things are there?), aboolean value (are there more blue things than green?), oranother scene (what if all of the red things turned blue?).This domain is modeled on CLEVR (Johnson et al., 2017a)but designed to support inductive synthesis tasks specifiedover the symbolic scene representations (an array of objectsrepresented as dictionaries of attributes) from the originalCLEVR task generator in Johnson et al. (2017a). We alsoadd new tasks that require generating or imagining latentscenes (how many metal things would be left if all the bluecylinders were removed?), which are not solvable in theoriginal high-level DSL hand-designed for Johnson et al.(2017b) (and used in synthesis-based approaches like Yiet al. (2018)). We include these to demonstrate a key fea-ture of our approach: the ability to learn generalizable li-braries from a basic but expressive set of primitives, ratherthan restricting the program space pre-emptively with ahand-designed language. We use synthetic language an-notations from the original templates in (Johnson et al.,2017a) (and templates written in the same style for the

extended tasks); human annotations are sourced from an-notators shown the same tasks. We initialize synthesizerswith functional programming primitives similar to the string-editing domain, with domain-specific query functions andconstants (get color(x); get shape(x); blue; cube). The neu-ral model encodes the task examples as flattened arrays ofobject attributes using a bidirectional GRU.

6.2. Results

On all three domains, we compare our model against thebaseline synthesizer (Table 1, DreamCoder, no language);a multimodal baseline (Table 1, multimodal, no genera-tive model) that trains a neural model directly on solvedtraining tasks (similar to neural synthesis models like Deep-Coder (Devlin et al., 2017) but augmented to condition onlanguage); and ablated LAPS variants (Table 1; LAPS rows)to evaluate the additive contributions of the individual learn-ing components. We compare all models using a matchedsearch budget per task and number of training iterationsoverall, determined using a hyperparameter search with thebaseline. The supplement contains full details (and code) toreplicate all experiments; and additional qualitative results.

We find that:

(1) LAPS searches more effectively during training, enablingit to solve and learn from more training tasks than the base-line synthesizer. Under the hierarchical model formulation,search and abstraction are closely related: successfully solv-ing tasks is the basis for abstraction learning.

Comparing the model learning trajectories (Fig. 3) on train-ing tasks shows that the LAPS models consistently searchmore effectively during training: at each iteration they solvemore tasks within a given time budget. Fig. 3 also highlightsthat LAPS models improve training robustness in the distantlearning setting: as in the baseline paper (Ellis et al., 2021),we find the baseline model learning to be highly variablewithout a training curriculum (compare training curves fromFig. 3 with different random seed replications; and the bestvs. mean performance, Table 1.) Comparing the LAPSablations also suggests that linguistic priors (like mutualexclusivity) can indeed be practically useful here duringlearning (Table 1, compare LAPS with ME and without).

What if we do use a curriculum? In the scene reasoningdomain (where previous approaches (e.g. Mao et al. 2019)have argued for a curriculum), we also test a simple cur-riculum by ordering tasks according to their natural lan-guage token length (which can be evaluated without groundtruth programs). Table 1 shows that our model is still moreeffective, and that non-curriculum performance is in factcomparable to curriculum performance.

(2) LAPS abstracts more effectively during training, addingin more generalizable library routines as it learns. The


Table 1. % held-out test-tasks solved. To compare robustness, we run random seed replications in the graphics domain for the syntheticlanguage dataset. Best reports the best model across replications; Mean averages across replications.Language Model Strings (ntest = 500) Graphics (ntest = 111) Scenes (ntest = 115)

% Solved % Solved (Best) % Solved (Mean) % Solved (Curric.) % Solved (Mean.)

Synth train/test DreamCoder (no language) 33.4 49.55 42. 64 67.80 73.9Synth train/test Multimodal (no generative translation model) 46.00 26.12 23.20 76.50 49.5

Synth train/test LAPS in neural search 52.20 92.79 52.93 95.6 88.1Synth train/test LAPS + mutual exclusivity 57.00 86.49 80.18 96.5 82.3Synth train/test LAPS + ME + language-program compression 54.60 98.19 81.98 95.6 95.9

Synth train/human test LAPS + ME + language-program compression 54.60 89.20 – 97.4 –Human train/human test LAPS + ME + language-program compression 48.60 58.55 – 95.6 –

No language at test

No language on train/test Original DSL; Enumerative 0.06 0.00 – 27.8 –No language on train/test DreamCoder (best library): Enumerative 27.2 41.44 – 53.6 –No lang at test LAPS (best library): Enumerative 33.2 62.16 – 93.04 –No lang at test LAPS (best library): example-only neural synthesis 52.4 91.0 – 95.6 –

LAPS + ME + lang. compression

% S

olve

d (0

–10

0%)

# Learning Iterations (0 – 27)

DreamCoder (no language) LAPS + mutual exclusivityLAPS in neural searchMultimodal (no generative)

Figure 3. Learning curves comparing baselines and LAPS models in Table 1, showing % heldout tasks solved on the graphics domainover random training task orderings. (Mean results in Table 1 shows average test-time performance from the trained model replications.)

variability across training replications in the baselines alsohighlights a challenge for abstraction learning: not all sharedsubroutines encountered in training generalize well to newtasks. Adding poor abstractions can actually be detrimen-tal: they increase the combinatorial search space. We findthat our approach produces higher-quality libraries aftertraining: Table 1 (no language at test time section) showsthat we consistently improve performance in a head-to-headcomparison using enumerative search from the library pri-ors alone – in some domains, enumerative search with ourmodel’s library outperforms neurally guided search fromthe baseline model. We also find the learned library iseffective for neurally-guided synthesis when no languagehints are available after training (Table 1, no language attest, example-guided synthesis), showing that LAPS in-corporates language to learn a more effective library overall,which generalizes to the non-language setting. See supple-ment for example learned abstractions from Lf .

(3) LAPS can use language during testing if it is avail-able, though it doesn’t need to for competitive performance.Clearly, language can provide a useful source of high-levelinformation if it is available for new tasks. Our approachproduces a neural synthesizer pre-trained to condition onlanguage where available. Results on all three domains showthat the model can use it to achieve additional performancegains (Table 1, see language at test rows). We also find thatthe models trained on synthetic annotations generalize effec-tively to natural human language at test (Table 1, synth train,human test), suggesting that even if human annotation is toocostly, in many cases hand-writing natural language tem-

plates to accompany a few ground-truth programs is likelysufficient (and easier than hand designing a full DSL).

7. ConclusionWe presented Language for Abstraction and ProgramSearch (LAPS). LAPS builds on hierarchical Bayesian mod-els of program learning: we offer a general framework forintroducing jointly generative models over programs andlanguage into learned synthesis. Going forwards, an impor-tant avenue for future work will be exploring different con-crete implementations of the base algorithm and translationmodel which relates programs to language. A promising fu-ture direction could leverage recent structured, neural jointmodels that can learn the compositional units of language,and incorporate pre-trained language representations (Joshi& Schabes, 1997; Wiseman et al., 2018; Kim et al., 2019).

The hierarchical Bayesian framing also draws connectionsto computational cognitive models which model human con-ceptual representations and learning (Goodman et al., 2014;Fodor, 1975; Rule, 2020) as inference over program-likerepresentations. Future human experiments could exploreLAPS as a cognitive model, combining paradigms for study-ing language learning with those for studying non-linguisticabstraction and search (e.g. Smith et al. 2003; Hawkins et al.2019; Lake et al. 2015; 2019; Tian et al. 2020).

Acknowledgements: Many thanks to M. Nye, J. Mu, A. Mar-zoev, J. Fan, R. Hawkins, R. Levy, L. Schulz and our anonymousreviewers for invaluable feedback. Supported by grants from theAir Force Office of Scientific Research, the NSF under Grant No.


ReferencesAbelson, H. and DiSessa, A. A. Turtle geometry: The

computer as a medium for exploring mathematics. MITpress, 1986.

Abolafia, D. A., Norouzi, M., Shen, J., Zhao, R., and Le,Q. V. Neural program synthesis with priority queue train-ing. arXiv preprint arXiv:1801.03526, 2018.

Andreas, J., Klein, D., and Levine, S. Learning with latentlanguage. arXiv preprint arXiv:1711.00482, 2017.

Appel, A. W., Beringer, L., Chlipala, A., Pierce, B. C., Shao,Z., Weirich, S., and Zdancewic, S. Position paper: thescience of deep specification. Philosophical Transac-tions of the Royal Society A: Mathematical, Physical andEngineering Sciences, 375(2104):20160331, 2017.

Artzi, Y. and Zettlemoyer, L. Weakly supervised learningof semantic parsers for mapping instructions to actions.Transactions of the Association for Computational Lin-guistics, 1:49–62, 2013.

Bahdanau, D., Cho, K., and Bengio, Y. Neural machinetranslation by jointly learning to align and translate. arXivpreprint arXiv:1409.0473, 2014.

Balog, M., Gaunt, A. L., Brockschmidt, M., Nowozin, S.,and Tarlow, D. Deepcoder: Learning to write programs.arXiv preprint arXiv:1611.01989, 2016.

Brown, P. F., Della Pietra, S. A., Della Pietra, V. J., andMercer, R. L. The mathematics of statistical machinetranslation: Parameter estimation. Computational linguis-tics, 19(2):263–311, 1993.

Chen, X., Liu, C., and Song, D. Tree-to-tree neural networksfor program translation. arXiv preprint arXiv:1802.03691,2018.

Cropper, A. and Muggleton, S. H. Learning efficient logicalrobot strategies involving composable objects. AAAIPress/International Joint Conferences on Artificial Intelli-gence, 2015.

Dechter, E., Malmaud, J., Adams, R. P., and Tenenbaum,J. B. Bootstrap learning via modular concept discovery. InTwenty-Third International Joint Conference on ArtificialIntelligence, 2013.

Delaware, B., Pit-Claudel, C., Gross, J., and Chlipala, A.Fiat: Deductive synthesis of abstract data types in a proofassistant. Acm Sigplan Notices, 50(1):689–700, 2015.

1918839 and NSF-funded Center for Brains, Minds, and Machines,the MIT-IBM Watson AI Lab, Google, Microsoft and Amazon.

Desai, A., Gulwani, S., Hingorani, V., Jain, N., Karkare,A., Marron, M., and Roy, S. Program synthesis usingnatural language. In Proceedings of the 38th InternationalConference on Software Engineering, pp. 345–356, 2016.

Devlin, J., Uesato, J., Bhupatiraju, S., Singh, R., Mohamed,A.-r., and Kohli, P. Robustfill: Neural program learningunder noisy i/o. In Proceedings of the 34th InternationalConference on Machine Learning-Volume 70, pp. 990–998. JMLR. org, 2017.

Dumancic, S. and Cropper, A. Inventing abstractions byrefactoring knowledge.

Ellis, K., Ritchie, D., Solar-Lezama, A., and Tenenbaum,J. B. Learning to infer graphics programs from hand-drawn images. arXiv preprint arXiv:1707.09627, 2017.

Ellis, K., Morales, L., Sable-Meyer, M., Solar-Lezama, A.,and Tenenbaum, J. Learning libraries of subroutinesfor neurally–guided bayesian program induction. In Ad-vances in Neural Information Processing Systems, pp.7805–7815, 2018.

Ellis, K., Nye, M., Pu, Y., Sosa, F., Tenenbaum, J., and Solar-Lezama, A. Write, execute, assess: Program synthesiswith a repl. arXiv preprint arXiv:1906.04604, 2019.

Ellis, K., Wong, C., Nye, M., Sable-Meyer, M., Cary, L.,Morales, L., Hewitt, L., Solar-Lezama, A., and Tenen-baum, J. Dreamcoder: Growing generalizable, inter-pretable knowledge with wake-sleep bayesian programlearning. ArXiv preprint, 2020.

Ellis, K., Wong, C., Nye, M., Sable-Meyer, M., Cary, L.,Morales, L., Hewitt, L., Solar-Lezama, A., and Tenen-baum, J. Dreamcoder: Bootstrapping inductive program-synthesis with wake-sleep library learning. PLDI 2021,2021.

Fikes, R. E. and Nilsson, N. J. Strips: A new approach tothe application of theorem proving to problem solving.Artificial intelligence, 2(3-4):189–208, 1971.

Fodor, J. A. The language of thought, volume 5. Harvarduniversity press, 1975.

Frank, M. C., Goodman, N. D., and Tenenbaum, J. B. Us-ing speakers’ referential intentions to model early cross-situational word learning. Psychological science, 20(5):578–585, 2009.

Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J.,Ranzato, M., and Mikolov, T. Devise: A deep visual-semantic embedding model. In Advances in neural infor-mation processing systems, pp. 2121–2129, 2013.


Gal, Y. and Blunsom, P. A systematic bayesian treatmentof the ibm alignment models. In Proceedings of the 2013Conference of the North American Chapter of the Associ-ation for Computational Linguistics: Human LanguageTechnologies, pp. 969–977, 2013.

Gandhi, K. and Lake, B. M. Mutual exclusivity as achallenge for deep neural networks. arXiv preprintarXiv:1906.10197, 2019.

Ganin, Y., Kulkarni, T., Babuschkin, I., Eslami, S. A., andVinyals, O. Synthesizing programs for images using rein-forced adversarial learning. In International Conferenceon Machine Learning, pp. 1666–1675. PMLR, 2018.

Goodman, N. D. and Frank, M. C. Pragmatic language inter-pretation as probabilistic inference. Trends in cognitivesciences, 20(11):818–829, 2016.

Goodman, N. D., Tenenbaum, J. B., and Gerstenberg, T.Concepts in a probabilistic language of thought. Tech-nical report, Center for Brains, Minds and Machines(CBMM), 2014.

Goyal, P., Niekum, S., and Mooney, R. J. Pixl2r: Guidingreinforcement learning using natural language by map-ping pixels to rewards. arXiv preprint arXiv:2007.15543,2020.

Grice, P. Studies in the Way of Words. Harvard UniversityPress, 1989.

Gulwani, S., Hernandez-Orallo, J., Kitzelmann, E., Muggle-ton, S. H., Schmid, U., and Zorn, B. Inductive program-ming meets the real world. Communications of the ACM,58(11):90–99, 2015.

Gulwani, S., Polozov, O., Singh, R., et al. Program synthesis.Foundations and Trends® in Programming Languages, 4(1-2):1–119, 2017.

Hawkins, R. X., Goodman, N. D., and Goldstone, R. L. Theemergence of social norms and conventions. Trends incognitive sciences, 23(2):158–169, 2019.

Jia, R. and Liang, P. Data recombination for neural semanticparsing. arXiv preprint arXiv:1606.03622, 2016.

Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L.,Lawrence Zitnick, C., and Girshick, R. Clevr: A diag-nostic dataset for compositional language and elementaryvisual reasoning. In Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition, pp.2901–2910, 2017a.

Johnson, J., Hariharan, B., Van Der Maaten, L., Hoffman,J., Fei-Fei, L., Lawrence Zitnick, C., and Girshick, R.Inferring and executing programs for visual reasoning.

In Proceedings of the IEEE International Conference onComputer Vision, pp. 2989–2998, 2017b.

Johnson, M. Pcfg models of linguistic tree representations.Computational Linguistics, 24(4):613–632, 1998.

Joshi, A. K. and Schabes, Y. Tree-adjoining grammars. InHandbook of formal languages, pp. 69–123. Springer,1997.

Kim, Y., Dyer, C., and Rush, A. M. Compound probabilisticcontext-free grammars for grammar induction. arXivpreprint arXiv:1906.10225, 2019.

Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B.Human-level concept learning through probabilistic pro-gram induction. Science, 350(6266):1332–1338, 2015.

Lake, B. M., Ullman, T. D., Tenenbaum, J. B., and Gersh-man, S. J. Building machines that learn and think likepeople. Behavioral and brain sciences, 40, 2017.

Lake, B. M., Linzen, T., and Baroni, M. Human few-shotlearning of compositional instructions. arXiv preprintarXiv:1901.04587, 2019.

Lau, T. A. and Weld, D. S. Programming by demonstration:An inductive learning formulation. In Proceedings of the4th international conference on Intelligent user interfaces,pp. 145–152, 1998.

Lazaro-Gredilla, M., Lin, D., Guntupalli, J. S., and George,D. Beyond imitation: Zero-shot task transfer on robotsby learning concepts as cognitive programs. ScienceRobotics, 4(26), 2019.

Liang, P. Learning executable semantic parsers for naturallanguage understanding. Communications of the ACM,59(9):68–76, 2016.

Liang, P., Jordan, M. I., and Klein, D. Learning programs:A hierarchical bayesian approach. In Proceedings ofthe 27th International Conference on Machine Learning(ICML-10), pp. 639–646, 2010.

Liang, W., Zou, J., and Yu, Z. Alice: Active learning withcontrastive natural language explanations. arXiv preprintarXiv:2009.10259, 2020.

Luketina, J., Nardelli, N., Farquhar, G., Foerster, J., Andreas,J., Grefenstette, E., Whiteson, S., and Rocktaschel, T. Asurvey of reinforcement learning informed by naturallanguage. arXiv preprint arXiv:1906.03926, 2019.

Mao, J., Gan, C., Kohli, P., Tenenbaum, J. B., and Wu, J.The neuro-symbolic concept learner: Interpreting scenes,words, and sentences from natural supervision. arXivpreprint arXiv:1904.12584, 2019.


Markman, E. M. and Wachtel, G. F. Children’s use of mutualexclusivity to constrain the meanings of words. Cognitivepsychology, 20(2):121–157, 1988.

Mu, J., Liang, P., and Goodman, N. Shaping visual represen-tations with language for few-shot classification. arXivpreprint arXiv:1911.02683, 2019.

Nye, M., Hewitt, L., Tenenbaum, J., and Solar-Lezama,A. Learning to infer program sketches. arXiv preprintarXiv:1902.06349, 2019.

Parisotto, E., Mohamed, A.-r., Singh, R., Li, L., Zhou, D.,and Kohli, P. Neuro-symbolic program synthesis. arXivpreprint arXiv:1611.01855, 2016.

Polosukhin, I. and Skidanov, A. Neural program search:Solving data processing tasks from description and exam-ples. 2018.

Polozov, O. and Gulwani, S. Flashmeta: a frameworkfor inductive program synthesis. In Proceedings of the2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Appli-cations, pp. 107–126, 2015.

Rule, J. S. The child as hacker: building more human-likemodels of learning. PhD thesis, Massachusetts Instituteof Technology, 2020.

Shin, E. C., Allamanis, M., Brockschmidt, M., and Polo-zov, A. Program synthesis and semantic parsing withlearned code idioms. In Advances in Neural InformationProcessing Systems, pp. 10824–10834, 2019.

Si, X., Yang, Y., Dai, H., Naik, M., and Song, L. Learninga meta-solver for syntax-guided program synthesis. InInternational Conference on Learning Representations,2019.

Silver, T., Allen, K. R., Lew, A. K., Kaelbling, L. P., andTenenbaum, J. Few-shot bayesian imitation learning withlogical program policies. In Proceedings of the AAAI Con-ference on Artificial Intelligence, volume 34, pp. 10251–10258, 2020.

Smith, K., Brighton, H., and Kirby, S. Complex systemsin language evolution: the cultural emergence of compo-sitional structure. Advances in Complex Systems, 6(04):537–558, 2003.

Srivastava, S., Labutov, I., and Mitchell, T. Joint conceptlearning and semantic parsing from natural language ex-planations. In Proceedings of the 2017 conference onempirical methods in natural language processing, pp.1527–1536, 2017.

Tian, L. Y., Ellis, K., Kryven, M., and Tenenbaum, J. B.Learning abstract structure for drawing by efficient motorprogram induction. arXiv preprint arXiv:2008.03519,2020.

Wiseman, S., Shieber, S. M., and Rush, A. M. Learn-ing neural templates for text generation. arXiv preprintarXiv:1808.10122, 2018.

Wong, Y. W. and Mooney, R. Learning synchronous gram-mars for semantic parsing with lambda calculus. In Pro-ceedings of the 45th Annual Meeting of the Associationof Computational Linguistics, pp. 960–967, 2007.

Ye, X., Chen, Q., Dillig, I., and Durrett, G. Benchmark-ing multimodal regex synthesis with complex structures.arXiv preprint arXiv:2005.00663, 2020a.

Ye, X., Chen, Q., Dillig, I., and Durrett, G. Optimal neuralprogram synthesis from multimodal specifications. arXivpreprint arXiv:2010.01678, 2020b.

Yi, K., Wu, J., Gan, C., Torralba, A., Kohli, P., and Tenen-baum, J. Neural-symbolic vqa: Disentangling reasoningfrom vision and language understanding. In Advances inNeural Information Processing Systems, pp. 1031–1042,2018.

Zhang, Y., Pasupat, P., and Liang, P. Macro grammars andholistic triggering for efficient semantic parsing. arXivpreprint arXiv:1707.07806, 2017.

Supplemental: Leveraging Language to Learn Program Search Heuristics andAbstractions

This contains the supplemental appendix to the 2021 ICML paper. It is organized sequentially in reference to the main text;S{N} refers back to section N in the main text.A complete release of code for our implementation, including command line scripts to replicate the experiments in the paperand links to the datasets, can be found at: https://bit.ly/3g9361W.

S4. Base learning algorithm: DreamCoderThe LAPS framework described in the main paper (Sec. 5)is a general one for extending Bayesian models of programlearning to incorporate information from natural language(see (Liang et al., 2010; Lake et al., 2015; Dechter et al.,2013; Lake et al., 2013)). Our concrete implementation andexperiments use the DreamCoder approach of (Ellis et al.,2021; 2018) as the base synthesis algorithm, which imple-ments the hierarchical Bayesian formulation of programlearning. It defines a modular interface with two primarylearning components: a learned conditional inference modelfor search (as a neural search heuristic); and a learned ab-straction algorithm for updating the program prior (basedon program refactoring and compression) (Ellis et al., 2021).Each of these learning components has been additionallyimplemented in other work (such as (Devlin et al., 2017;Polosukhin & Skidanov, 2018; Nye et al., 2019; Parisottoet al., 2016; Balog et al., 2016) for neurally guided synthesis,and (Dechter et al., 2013; Zhang et al., 2017; Shin et al.,2019; Artzi et al., 2014; Dumancic & Cropper) for programabstraction learning).

This supplementary section provides theoretical and im-plementation details on the DreamCoder algorithm we usein our experiments (summarized in Sec. 4). We matchour implementation as closely as possible to the originalwork for comparison with published baselines. We providekey details relevant to the language-guided extension, butstrongly recommend the original works which introduce theDreamCoder algorithm (Ellis et al., 2021; 2018) for furtherreference.

S4.1 Program prior and MDL equivalence

Hierarchical Bayesian program learning formulations re-quire a prior over expressible programs. DreamCoder islearned iteratively: it is initialized with a base library L0

and returns a library Lf containing program abstractionslearned from solving training tasks. Therefore, Dream-Coder defines its program prior with respect to the currentlibrary Li maintained at each iteration. This is parame-

terized as a simple PCFG P[ρ|L, θL] whose productionsare of the form li → lj ∈ L, each with a real-valuedweight θLl, where the probability of a program ρ is givenby P[ρ|L, θL] =

∏l∈ρ P[l|L, θL] (Sec. 4.1).

Minor complexity arises in order to support typing (Pierce,2002): following (Ellis et al., 2018), the library Li is im-plemented as a set of polymorphically typed λ-calculusexpressions. The only change this produces to the originalprior definition is to restrict the set of possible productionsunder the PCFG: that is, permissible productions are of theform li → lj ∈ {L|li → lj is well typed}. The prior proba-bilities of programs are therefore calculated with respect tothe set of well-typed productions.

As discussed in the main paper, this prior definition is equiv-alent to a minimum description-length prior over programsunder (L, θL) when all θL < 1.0, as the product of addi-tional productions in an expression will strictly decrease asthe number of productions in an expression increases.

S4.2 Amortized conditional inference

Figure 1. Architecture of the neural modelQi(ρ|t,Li). The modeltakes as input task examples t. These are encoded using a domain-specific encoder E(t). Task encodings feed to an MLP and activa-tion layer and output a tensor Q. This parameterizes a distributionover program bigrams in the final DSL, which defines a conditionaldistribution from which to enumerate programs during search.

To identify programs that solve tasks t while obtaining highprobability under P[ρ|L, θL], DreamCoder trains a neuralsearch heuristic Qi(ρ|t,Li) at each iteration i to approxi-mate the inverse model.

The training procedure in (Ellis et al., 2021) (summarized inSec. 4.2) is a key contribution of the original work for learn-ing in the distant supervision setting. The model is trainedon samples from the generative prior (providing an endless

arX

iv:2

106.

1105

3v3

[cs

.LG

] 3

May

202

2

Supplemental: Leveraging Language to Learn Program Search Heuristics and Abstractions

training stream of random synthesis tasks); and this proce-dure should generalize immediately to any neural model forpredicting programs conditioned on the task specification(e.g. (Devlin et al., 2017; Polosukhin & Skidanov, 2018;Nye et al., 2019; Parisotto et al., 2016; Balog et al., 2016)).The model is also supervised on any original training taskexamples and their program solutions discovered duringlearning.

In our experiments we use the baseline neural model archi-tecture in (Ellis et al., 2021). This is parameterized by twomodular components:

1. A domain-specific task encoder E(t). This encodes thetask examples (e.g. images in the graphics program do-main, or input-output strings in the text editing domain)that are input to the neural model. This task encoder ar-chitecture is defined domain-specifically based on theform of the task examples (e.g. a CNN for the graphicsdomain). It outputs a fixed dimensional embedding forany given task as input to the model. In our experi-ments this is a 64-dimensional embedding across alldomains (See S6.1 for domain-specific architectures;and released code.)

2. A conditional model over programs Q(ρ|E(t)). Thiscomponent receives the task encoding as input andoutputs a distribution over programs. Following (Elliset al., 2021), this is a 2-layer fully-connected MLP(with 64 hidden units and a final tanh activation layer)that outputs a fixed-dimensional real-valued tensor en-coding a distribution over programs in the library L asoutput. The real-valued tensor corresponds to weightsover program primitives conditioned on their local con-text in the syntax tree of the program, consisting of theparent node in the syntax tree and which argument isbeing generated. This functions as a ‘bigram transitionmodel’ over trees that encodes the likelihood of transi-tions from one primitive to the next. Q returns this as a(|L|+ 1)× (|L|+ 2)×A-dimensional tensor, whereA is the maximum arity of any primitive in the library.

This parameterization supports fast sampling of programsduring conditional synthesis: the neural model runs once pertask (to encode the task examples and produce the bigramtransition model) and the resulting parameterization canthen be used to sample programs during synthesis (e.g. byenumerating programs by expanding trees (as ‘bigrams’over parent and children primitives) ranked in order of theirlikelihood starting from the program root.)

Following (Ellis et al., 2021), the neural model is trainedto optimize the following MAP inference objective on the

training tasks and the sampled tasks from the prior:

LMAP = Et∼(L,θL)

[logQ

(argmax

ρP[ρ|t,L, θL]

∣∣∣∣ t)]

(1)

S4.3 Abstraction learning as program compression

DreamCoder learns new abstractions to approximately opti-mize for Eq. 2 (main paper), which infers an optimal libraryand parameters with respect to the observed programs onthe training tasks.

The DreamCoder abstraction algorithm is a primary con-tribution of the original work in (Ellis et al., 2021), and isdiscussed extensively in (Ellis et al., 2021). We thereforeprovide additional technical details here that are relevant toits integration with LAPS in our experiments, but stronglyencourage referencing (Ellis et al., 2021) for the full imple-mentation.

As discussed in (Ellis et al., 2021) and our main work,DreamCoder approaches abstraction using an equivalencebetween Eq. 3 and the minimum description length of theprior (as the description length of the library) and the pro-grams produced from the prior (under the PCFG definitionof the prior). Therefore, in practice, inferring the optimal li-brary is equivalent to inferring the library which maximallycompresses the description length of the library and thedescription length of programs which explain the trainingtasks. In particular, DreamCoder optimizes the followingcompression objective with respect to the training tasks Tand the finite beam Bt of program solutions discovered foreach training task during learning:

log P[L] + argmaxθL

∑

t∈Tlog

∑

ρ∈Bt

P[t|ρ] maxρ′−→∗ρ

P[ρ′|L, θL]

+ log P[θL|L]− |θL|0 (2)

The key aspect of this algorithm is that it considers abstrac-tions which compress not only the programs as they are cur-rently written, but any semantically equivalent refactoringsof these programs. Specifically, as programs are written in aλ-calculus, refactoring refers to any program which is equiv-alent up to β-reduction (i.e., function application/variablesubstitution (Pierce, 2002)). A primary contribution of theoriginal work in (Ellis et al., 2021) is an efficient algorithmfor computing these refactorings that is unchanged when weintegrate language; we refer to the original text for details.

In our work, the primary important aspect of this aspect isthat refactorings are defined compositionally over the ex-isting program primitives. Specifically, refactorings can beefficiently calculated according to semantic equivalencesin the the λ-calculus (namely, that function application andvariable substitution guarantee that the resulting refactoredprograms are equivalent. Abstractions created by variable


substitution will always be composed of subcomponentsfrom the initial library.) We take advantage of this composi-tionality when defining our joint abstraction algorithm overnatural language. Defining an initial compositional transla-tion model between language and the program componentsensures that we can approximate compression in the jointmodel after the programs are refactored, without needing toinduce an entirely new translation model over language andthe refactored programs.

S5. Our Approach: Language for Abstractionand Program SearchThis section now describes technical details for the concreteLAPS implementation in our reported experiments, whichis defined over the DreamCoder implementation. We struc-ture this section according to the parallel implementationsin the base algorithm for clarity. However, except for thespecifics of the joint-abstraction algorithm, the technicalimplementation of each component should extend directlyto most other similar learned synthesis algorithms (e.g. thejoint model implementation should be reusable in any syn-thesis algorithm that uses an explicit symbolic library ofprimitives.)

S5.1 Joint prior over programs and language

LAPS extends the prior P[ρ] over programs under the libraryto a joint prior J(ρ, dt) over programs for a given taskand their natural language descriptions dt (Sec. 5.1). Weformulate this prior as

J(ρ, dt) = P[ρ|L, θL]P[dt|ρ,L]

the product of the original prior over programs P [ρ|L, θL]defined on the program library, and a program to descrip-tions “translation” model T (dt|ρ,L) ≈ P[dt|ρ,L] that de-scribes how descriptions are generated for programs writtenin the library.

The concrete implementation described in the main paperuses a translation model that additionally decomposes com-positionally over language and programs–in particular, onthe basis of token-token translation distributions PT [w|l]between words w ∈ dt and l ∈ L. Many available trans-lation and semantic parsing models (such as synchronousgrammars over natural language and programs) preservethis further compositional requirement (e.g. (Artzi et al.,2014; Wong & Mooney, 2006)).

See Figure S3 (supplement) for example samples from thegenerative model on the graphics domain at earlier and laterstages of training.

Our implementation uses a classical statistical machinetranslation model (the Model 4 version of the IBM Statis-

tical Machine Translation models (Gal & Blunsom, 2013))whose parameters can be tractably estimated from very fewpaired programs and descriptions (in the distant supervisionsetting used in the original work, there may be no morethan a couple of hundred training tasks in the full dataset,and fewer than 10 solved tasks on which to train the trans-lation model at any given time.) In addition to inferencein small data settings, this translation model has a fullycompositional generative definition (Gal & Blunsom, 2013)that allows it to be easily used to train the neural amortizedinference model which conditions on language.

Despite this, however, this translation model (and the furtherinductive biases used to specifically relate program trees tosentences) make strong compositonality assumptions aboutthe relationship between program primitives and words asa joint generative model of programs and language; wefind that these inductive biases are useful in the small datasetting and produce empirically successful results. However,this is likely because of how the joint model is used duringtraining, which does not require a perfect generative modelof language (or language with respect to programs) for eitheramortizing inference or abstraction in order to use languageas a heuristic during learning.

A full definition of the statistical translation model we usecan be found in (Gal & Blunsom, 2013). We re-summarizeimportant details here. The IBM family of translationmodels estimates the conditional token-token probabilitiesPT [w|l] on the basis of alignment variables al,d, which spec-ify a direct correspondence between tokens in parallel texts(e.g. a word in a task description and a program primitive.)These alignments are many:many between tokens in pro-grams and natural language sentences – a given word cancorrespond to multiple primitives, and vice versa. Condi-tioned on a set of alignments from paired programs anddescriptions, the conditional probabilities in both directions(the probability of generating a program primitive in a pro-gram based on the presence of a word in a sentence, andvice versa) are defined by marginalizing over the alignmentvariables. We provide one direction (PT [w|l]), as the otheris symmetrical:

PT [w|l] ∝∑

a1

...∑

am

P[w, a1...am|l] ∝m∏

i=1

q(ai|i, l,m)

where ai are alignment variables inferred over a paired cor-pus and q(j|i, l,m) can be interpreted as the probabilityof alignment variable ai (for the token with index i in aprogram) taking value j (where j is an index into the corre-sponding sentence) conditioned on the lengths l and m ofthe program and natural language sentence (Gal & Blunsom,2013).

These alignments are inferred by approximately invertingthe generative model in (Gal & Blunsom, 2013) to maxi-


mize the likelihood of the observed paired sentences andprograms. One implementation detail: the alignment algo-rithm operates over pairs of strings. For convenience weinfer alignments between sentences and linearized tokensequences in the program tree (which can be done with com-plete recoverability of the original program tree (Andreaset al., 2013)). This is another inductive assumption that wechoose after preliminary experimentation and find that ourimplementation yields strong empirical results regardless.

The IBM translation model is a noisy-channel generativemodel that requires an additional language model p(d) togenerate language (Gal & Blunsom, 2013; Heafield, 2011).We use an efficient parallelized implementation for inferringthe translation model parameters from (Koehn et al., 2007),which also contains a basic language model inferencealgorithm inferred over the full corpus of training tasksentences (as a trigram model, which we again find simplebut effective for our very small data setting). Specific modelhyperparameters for all experiments are available in thereleased code repo (in the experiment runtime commands.)

Mutual exclusivity: Section 5.1 of the main paper alsodescribes how the joint model can be modified to includelanguage-specific priors, such as a simple implementationof the well-known mutual exclusivity prior documentedin the cognitive language-learning literature (Markman &Wachtel, 1988; Gandhi & Lake, 2019) and given a Bayesianformulation in (Frank et al., 2009). We provide an imple-mentation to demonstrate that the joint model can be easilyextended: specifically, a simple mutual exclusivity assump-tion can be added into the joint model by simply updatingthe compositional translation model to include additionaldistributions tME(dnew|l) where dnew are words that onlyappear in unsolved training tasks and

tME(dnew|l) ∝ αP[l|L, θL]−1

new words are now assumed to correspond to primitives in-versely proportional to their current usage under the learnedprogram prior. As we show in the next section, incorporat-ing this prior at the level of the joint model can be used toapproximate mutual exclusivity assumptions in the learnedsearch heuristic, encouraging exploration in the presence ofnew words.

Practically, we calculate the mutual exclusivity prior in ourconcrete implementation by leveraging the alignments uponwhich our token-token translation probabilities are defined.Specifically, we add pseudoalignments between each dnewand each l ∝ αP[l|L, θL]−1; when the token-token transla-tion probabilities marginalize over the latent alignments andthese pseudo alignments, the resulting translation probabili-ties encode the mutual exclusivity prior.

S5.2 Integrating the joint model into amortizedconditional search

Figure 2. Architecture of the language-conditioned neural modelQ(ρ|d, t). The model takes as input task examples t. These areencoded using a domain-specific encoder E(t). The model ad-ditionally takes in task descriptions d, encoded using a languagencoder ED(t) (implemented as a GRU). Task encodings are con-catendated and feed to an MLP and activation layer and output atensor Q. This parameterizes a distribution over program bigramsin the final DSL, which defines a conditional distribution fromwhich to enumerate programs during search.

The amortized conditional inference model Q(ρ|t) (Sec.4.2) extends straightforwardly in LAPS to condition on lan-guage Q(ρ|d, t) (Sec. 5.2). Importantly, the training proce-dure in Sec. 4.2 (training the neural model on samples fromthe prior) also extends to the language-enriched condition(training the neural model on samples from the joint prior,which include generated language annotations.)

In our experiments we implement the concrete neural modelQ(ρ|d, t) in our experiments by extending modularly on theoriginal model in (Ellis et al., 2021) (and in the supplementalS4.2) for direct comparison. Our full architecture thereforehas three modular components to additionally condition onlanguage:

1. A natural language task descriptions encoder ED(d).This receives the task description d as input. We imple-ment this as an RNN model using a bidirectional GRU(Cho et al., 2014) with 64 hidden units; we embednatural language symbols as 64-dimensional vectors,and randomly initialize and backpropagate through theembedding during training. We tokenize the sentencesin u on whitespace and concatenate each sentence, de-limited by special start and end of sentence tokens. Attest time, we replace any OOV tokens with a specialUNK token.

2. A domain-specific task encoder E(t), following S4.2.

3. A bigram transition model over program primitives,following S4.2. To condition jointly on ED(d) andE(t) we simply concatenate these two embeddingsand update the first layer of the MLP to take the 128-dimensional concatenated embeddings as input.


5.3 Abstraction learning as joint model compression

Finally, the abstraction learning model in (Ellis et al., 2021)can also be generalized to condition on language, by extend-ing the optimal library inference algorithm with respect tothe program prior to an optimal library inference algorithmwith respect to the joint model over language and programs(Eq. 6 and 7, main text.)

In our concrete implementation with respect to the Dream-Coder algorithm, this means extending the description-length compression objective – originally defined over theprogram library and training task programs – to includethe translation model definition. The main paper defines adescription-length prior over the compositional translationmodel (Eq. 10). Optimizing this tractably requires redefin-ing the abstraction algorithm in (Ellis et al., 2021) – whichrefactors λ-calculus programs via lambda-abstraction (seeS4.3 for a summary) – to also jointly re-estimate the descrip-tion length of the translation model T (dt|ρ,L′) using therefactored programs under the new candidate library L′.We implement an efficient approximation that can be cal-culated with respect to the classical statistical translationmodel described in S4.1 (Gal & Blunsom, 2013). In particu-lar, we leverage the alignment-based definition (which useslatent correspondences inferred between program tokensand sentence tokens in paired programs and descriptions) toapproximate −H(PT [w|l]) = − log(PT [w|l]), the entropyof the token-token translation probabilities.

Specifically, as the IBM model defines the conditional token-token probabilities

PT [w|l] ∝∑

a1

...∑

am

P[w, a1...am|l]

marginalized over alignments, where (slightly abusing nota-tion) in any given paired program and sentence descriptionwe will have estimated a set of alignments awj ,lk...ln be-tween the j-th token in the description corresponding to oneor more tokens lk...ln in the paired program. We thereforedefine the description-length of each token-token transla-tion as the sum of the description lengths of the alignmentswhich express it under a library L:

∑

ai

...∑

am

P[d, a1...am|l,L] ∝∑

a1

...∑

am

|ai|L

and the description lengths under the refactored library L′containing new abstractions compresses according to

|a′wj ,l′k...l′n|L′ < |a′wj ,lk...ln

|L ⇐⇒{l′icontains only lk...ln as subcomponents|l′k...l′n}

(3)

and we say that a primitive l ∈ L is a subcomponent ofa refactored abstraction l ∈ L if the abstraction can be

β-reduced such that l appears in it. That is, a refactoredalignment a′ : wi → {l′...ln} is compressed only when anew abstraction l′ encapsulates over a strict subset of theconstituent program primitives already aligned to the wordin the original alignment. This allows us to re-approximatethe description length of the new translation model withrespect to a semantically-equivalent program refactoringwithout inducing PT [w|l] from scratch (which would requireretraining the full translation model over the sentences andrefactored programs.)

S6. ExperimentsThis section describes additional details on each of the do-mains – string editing, compositional graphics, and sceneunderstanding – in Section 6 of the main paper (see Figure2, main text for examples from all three domains, shownalong with the synthetic and human language annotations).We also provide additional details on the model and baselinehyperparameters available for each domain. All datasetsgenerated for these experiments (including human languageannotations) are released and links to static repositories areprovided in the code release. We also release a complete setof commands to exactly replicate all model experiments.

All experiments for were conducted on a high-powered com-puting cluster using a fixed training budget of wall-clocksearch time per task for all models and baselines in a givendomain (determined via hyperparameter search using thebaseline model per domain, and reported on a per-domainbasis below). The experiments on the string editing andgraphics domains used models trained using 48 CPUs forsearch (using the original parallel enumerative search imple-mented in the released code for the DreamCoder model in(Ellis et al., 2021)); and the experiments trained on the scenereasoning task used 24 CPUs (as preliminary experimentsrevealed that these experiments required shorter search timefor our main model, and we wished to reduce the carbonfootprint of the remaining experiments after our first twodomains.)

For all experiments we train the neural models for 1 ×104gradient steps. For experiments with language-guided com-pression, we use an upper bound of 5 new abstractions in-troduced per iteration. For mutual exclusivity experiments,we set αME = 0.1. For all experiments, during program-only compression (see (Ellis et al., 2021) for a discussionof program-only compression hyperparameters) we use thehyperparameters from (Ellis et al., 2021) for parsimony withearlier work: a structure penalty of 1.5 and pseudocounts =30.


S6.1 Domains

(See Figure 2, main text for examples from all three do-mains, shown along with the synthetic and human languageannotations.) As discussed in the main paper, each domainconsists of a dataset of tasks; a set of procedurally generatedsynthetic language annotations; and a set of human lan-guage annotations provided by Mechanical Turk workers;we also described the base primitives L0 with which allmodels (including baselines and ablations) were initializedfor each domain.

S6.1.1 STRING EDITING

Tasks: structured string transformation problems takenfrom a publicly released dataset in (Andreas et al., 2017)(n=1000 train; n=500 test). Tasks consist of input dictio-nary strings transformed using randomly sampled regularexpression transducer (n=30 examples per task). Transduc-ers were sampled according to abstract templates definedin (Andreas et al., 2017) and required identifying matchedsequences of characters and adding letters before them; re-moving sequences; replacing them with new sequences, ordoubling the sequence each time they appeared (See Figure2A, main text).

Language data: The human language dataset for this do-main was previously collected by (Andreas et al., 2017). Wedefined a synthetic grammar of high-level templates over theground truth regular expression transducers (correspondingto the original templates used to generate the tasks.) Thesynthetic templates were defined based on language fromthe original human annotations, and in most cases closelymatched the true human provided annotations (which weregenerally quite structured), though with significantly lessvariation (the original language contained multiple humandescriptions per task. We generate a single synthetic foreach one. The synthetic dataset has a vocabulary size ofn=44 for both train and test. We use the human annota-tions in the original dataset when evaluating on human data,which have a vocabulary of n=727 (train) and n=622 (test).)We generate a synthetic dataset on this domain partly be-cause of inaccuracies noted in (Andreas et al., 2017). Thereleased code contains the complete generation procedurefor these synthetic annotations. See Figure 2A for represen-tative tasks with examples, synthetic language, and humandescriptions.

Initial program primitives: We initialize all models witha set L0 of LISP-like primitives that operate over substringsequences to both construct regular expression match se-quences and manipulate strings, augmented with three textmanipulation-specific primitives intended for executing con-structed regular expression sequences; t is a polymorphictype variable using standard Hindley-Milner polymorphismtyping (Pierce, 2002). The execution engine does include

a regex-matching model; however, the synthesis model isnaive to this execution engine and simply searches for ma-nipulations over the input strings and the regexes as dataarrays.

L0 contains 14 substring manipulation primitives, givenbelow with type information. We also give a semantic glossfor primitives that are not standard LISP primitives.

• if (bool → t → t → t)

• cons (t → list(t) → list(t))

• car (list(t) → t)

• cdr list(t) → list(t

• map ((t0 → t1)→ list(t0)→ list(t1))

• tail (list(t) → t)

• append (t → list(t) → list(t))Appends element to end of list.

• revcdr (list(t) → list(t))Takes all except the last element of the list.

• match (substr → substr → bool)Returns true if the first argument, when executed as aregular expression, matches the second argument.

• regexsplit (substr → fullstr →list(substr))Attempts to execute the first argument as a regularexpression, and splits the second argument into a listof substrings, using the regular expression match as adelimiter (and includes the matched sequences in thereturned list.)

• flatten (list(substr) → fullstr)Flattens a list of substrings back into a string.

• rconcat (substr → substr → substr)Concatenates two substrings.

• rnot (substr → substr)Takes a substring argument s and returns the substringliteral [ˆ s]

• ror (substr → substr → substr)Takes substring literals a and b and returns the substringliteral ((a)—(b))

We also include 26 character constants of type substr andconstants dot (regular expression wildcard character) andempty (empty string).

Domain hyperparameters We largely follow prior work(Ellis et al., 2021) to set algorithm training parameters; the


earlier (Ellis et al., 2021) uses a 720s enumerative searchbudget for solving both text editing and general list manip-ulation tasks. We use the same 720s enumerative budgethere.

The encoder E(t) follows the domain-specific encoder usedfor text and list editing problems in (Ellis et al., 2021), a2-layer GRU with 64 hidden units. The model is trainedfor a fixed gradient step budget (10,000 gradient steps) andwe sample equally at random between supervision on thesolved training tasks (and their solution programs in thecurrent DSL) and samples from the joint generative model.As with (Ellis et al., 2021), when generating tasks from thegenerative model, we use randomly sample inputs (on whichwe execute generated programs to produce an output.)

S6.1.2 COMPOSITIONAL GRAPHICS

Tasks: inverse graphics problems (n=200 train; n=111 test)where each synthesis problem is specified by an image andsolved by synthesizing a program in LOGO Turtle graph-ics (Abelson & DiSessa, 1986). The domain is inspired bythe graphics domain in (Ellis et al., 2021) but intentionallyre-designed to be much more challenging (ground-truth pro-grams are much longer on average in the base programminglanguage) and explicitly compositional: the training andtesting tasks contain simple shape tasks defined by composi-tional parameters for a set of basic shapes (a small triangle,a medium square; a small semicircle); complex shape tasksthat require inferring more challenging (and longer) param-eterized shapes (a greek spiral with eight turns); and compo-sitional tasks defined by geometric rules and relations overthe simple shapes (a seven sided snowflake with a short lineand a small triangle as arms; a small triangle connected bya big space from a small circle) (See Figure 2C).

Simple parameterized shapes are either polygons (triangle,square, [n] gon), curves (semicircle, circle) or lines. Simpleshapes are parameterized by one of three sizes (small orshort; medium; and big). When generating synthetic lan-guage descriptions, pluralized objects are tokenized withseparate tokens for the noun lemma and a token for the plu-ral suffix (e.g. square s).Complex parameterized shapes require constructing morecomplex images out of basic lines, and are intended to evalu-ate performance on tasks that pose a greater search challengein the initial DSL, and whose structure is not directly cuedby compositional relationships over easier components. Fur-ther, the complex shapes can be solved using abstractions(e.g. for repeatedly rotating a pen at right angles) that arenot directly cued by shared lexical names – we evaluate thealgorithm’s ability to learn and use abstractions that corre-spond to useful sublexical structures shared across multiplelexemes. We define four template families for complexshapes: spirals, staircases, zigzags, and stars.

Compositional graphics tasks invoke compositional rela-tionships over the simple parameterized shapes. We definetemplates for generating 6 families of compositional tasks:nested, next to, separated by, connected by, in a row, andsnowflakes.

Language data: We gather human language annotationsby asking Mechanical Turk workers to write an image de-scription for the rendered graphics images that specify eachtask. Each worker labeled 20 training and 10 testing imagesafter viewing a disjoint, randomly sampled set of 15 exam-ple images paired with their synthetic language captions.(Workers were asked to write a short, clear description thata person or robot could use to recreate the picture, andtold that the examples were paired with automatically gen-erated captions as an example of the kinds of descriptionsyou could write for this picture.) We control for descriptionquality by requiring workers to complete a reference task ontheir own descriptions: after writing their initial annotations,workers were required to correctly match each annotation tothe target image (from amidst a set of 12 distractors drawnheuristically from similar images on the full task dataset,and other images they themselves had described), and onlyannotations correctly matched to the target image were re-tained (workers were given a chance to redescribe picturesthey failed to match to their own captions.) We preprocessthe human dataset minimally to standardize number terms(e.g. we use the same token type for both 3 and three) andto split plurals into a lemma and suffix, as in the syntheticdataset. The final dataset has a vocabulary size of n=562 forboth train and test.

As with the string editing domain, we define a syntheticdataset using parameterized templates based on systematiclanguage reused in the human annotations (see Figure 2A fora comparison between human annotations and synthetic lan-guage); as with that domain, we choose a synthetic datasetto ensure systematic re-use of high level terms for repeatedcompositional objects (such as the “n-gon” or “snowflake”terminology.)

We then generate graphics tasks by defining parameterizedtemplates over ground truth programs in L0, and a corre-sponding generator for synthesizing natural language de-scriptions based on each ground truth program. It is impor-tant to note that the templates are defined at any extremelyhigh level and were written with respect to low-level pro-grams in a simple graphics language (many of which werederived by generalizing compositionally over complex struc-tures in (Ellis et al., 2021), such as the ‘snowflake’ images).

Initial program primitives: For comparison with priorwork, our initial library on this domain (and the base lan-guage used to generate the ground truth graphics programs)is an implementation of the LOGO Graphics DSL used in(Ellis et al., 2021), which consists of four typed, impera-


tive primitives modeled within the λ−calculus with a statemonad S:

move: distance → angle → S → Spen-up: (S → S) → S → Sfor: int → (S → S) → S → Sget/set: (S → S) → S → S

as well as four arithmetic operators (+, -, *. /), integerconstants (1-9), unit distances and angles (1 meter and 2πradians), and special values∞ and ε.

Figure 3 (main text) shows examples of the graphics tasks,synthetic descriptions, human descriptions, and sample pro-grams in the ground truth initial DSL.

Domain hyperparameters We largely follow prior work(Ellis et al., 2021) to set algorithm training parameters. Con-sistent with the graphics program experiments in (Ellis et al.,2021), we train all models, including baselines and abla-tions, using an enumerative search budget of 1800s per task(both when using pure enumerative search from the DSLprior, and neurally-guided search conditioned on the taskexamples and language descriptions); the results in Table1 compare the relative advantage of our model given thisfixed search time. We train all models on 48 CPUs dur-ing parallel enumerative search, and run the algorithm fora maximum of 27 iterations (see learning curves. As werun multiple random seed replications of models in this do-main, we tuned the iteration limit based on performance onthe first replication, allowing models models to train whileperformance continued to increase. To conserve computa-tional resources, we later stopped several of our own modelreplications before 27 iterations, as they had reached nearceiling performance. As we report the best held-out testscore across all 27 iterations for any one model, the earlystopping would only serve to give a conservative estimateon performance for these models.) We randomly reorder thetraining set of tasks once before the first loop, then iteratethrough batches of n=40 tasks at each iteration; learningcurves show results from evaluating on held-out tasks everyn=3 iterations.

The encoder E(t) follows the domain-specific encoder usedfor the original graphics domain in (Ellis et al., 2021) fora more direct comparison: we use a 6-layer CNN, whereeach layer consists of a 64x64 2D convolutional sublayerwith kernel size = 3, a RELU activation sublayer, and a max-pooling sublayer with kernel size = 2. The model is trainedfor a fixed gradient step budget (10,000 gradient steps) andwe sample equally at random between supervision on thesolved training tasks (and their solution programs in thecurrent DSL) and samples from the joint generative model.

S6.1.3 SCENE REASONING

Tasks: inductive scene reasoning tasks (n= 212 train; n=115test) where each synthesis problem is specified by a struc-tured input scene, and outputs can be a number (how manyred rubber things are there?), a boolean value (are theremore blue things than green things?), or another scene (whatif all of the red things turned blue?). This domain is modeledon CLEVR (Johnson et al., 2017) but designed to supportnon-linguistic, inductive synthesis in the programming-by-example paradigm: each task is specified with n=7 pairedinput output examples. See Figure 2B, main text for exam-ple tasks showcasing the original and extended templates,synthetic language annotations, and human language anno-tations.

The dataset includes questions randomly generated from thefollowing subset of the original CLEVR question templates(see (Johnson et al., 2017) for additional details on the taskgeneration process and question templates; we also releaseour own augmented question generation code and the fulldataset):

• zero hop: questions that require counting or answer-ing an attribute query about a subset of objects in thescene. (e.g. How many small cylinders are there?;What material is the purple thing?).

• one hop: questions similar to the zero hop tasks, butthat require reasoning over an additional relationalquery (e.g What number of things are right the smallgray thing?).

• single or: questions that additionally introduce a dis-junction between sets of objects. (e.g. How manyobjects are either large metal spheres or large rubberthings?)).

• (compare integer: questions that additionally intro-duce a ≥ or ≤ operator between counts of sets of ob-jects. (e.g. Is the number of large rubber cubes lessthan the number of large green rubber things?)

• same relate: questions that additionally require rea-soning about other objects with the same attribute asa specified object. (e.g. How many other things arethere of the same size as the cyan thing?).

We choose these templates as a representative subset ofthe style of the full CLEVR dataset, that requires the fulllanguage of high-level primitives in (Johnson et al., 2017)to solve. We omit some longer questions in the same format(e.g. two hop) as our intention is to compare synthesisbaselines, rather than to achieve SOTA performance onCLEVR: this would likely only increase the computingresources needed to compare the various methods and we


already found a significant differential between our modeland the baselines on the shorter questions.)

We also add new question templates generated in the styleof the original CLEVR tasks, but designed to model othercommon AI tasks (such as generating new scenes based onexisting ones) and to require new abstractions (that werenot expressible in the original restricted symbolic languageused to generate scenes in (Johnson et al., 2017)):

• localization: questions for object localization. Thesereturn an output scene consisting of a localized set ofobjects based on a set of query attributes (e.g. Find thegray rubber thing.).

• remove: questions that either return an output scenewith a subset of the objects removed, or that queryabout latent scenes where a subset of objects has beeremoved. (e.g What if you removed all of the graymetal things?; If you removed the green cubes, howmany cubes would be left?).

• transform: questions that either return an output scenewhere a subset of the objects has been transformed toset new attributes, or that query about latent sceneswhere a subset of objects has been modified this way.(e.g What if all the blue metal things became rubberthings?; If all of the large yellow rubber things becamegray spheres, how many gray spheres would there be?).

We treat these as program synthesis tasks: the input scenesare specified as symbolic scene graphs consisting of an ar-ray of structured, objects defined as a dictionary of theirattributes, and programs are designed to manipulate thesestructured arrays (this data structure is the original formatin which scenes themselves are generated in (Johnson et al.,2017); the images displayed in Figure 3, main text are ren-dered using the original image rendering pipeline). Our in-tention is not to build a visual reasoning architecture: rather,we are interested in learning structured manipulations ofscenes. We see work in inverse graphics (such as (Yi et al.,2018)) which outputs a structured scene graph based onpixel images as the first step in a symbolic processing andreasoning pipeline as analogous; we are interested in thestructured manipulation of these scene representations.

Language data: Synthetic language annotations are gener-ated based on the original high-level templates in (Johnsonet al., 2017), as well as additional templates we define forthe extended questions in the same style. We gather humanlanguage annotations by asking Mechanical Turk workersto write an instruction or question describing the set of in-ductive examples. However, due to the difficulty of solvingcertain tasks in a limited time frame based on the inductiveexamples alone (such as the questions about disjunctions

over scenes), we show Mechanical Turk workers the syn-thetic descriptions for this domain and ask them to write asemantically similar description that changes more than oneword in the original caption, and that would be ”more natu-ral for a human to understand”. This paraphrasing paradigmis similar to that used in (Wang et al., 2015), though we findthat in comparison to other domains it generates less diverselanguage data.) We remove all punctuation, tokenize onspaces, and use an additional domain heuristic to stem allplurals (e.g. cubes).

Initial program primitives: We initialize all models witha set L0 of LISP-like primitives. These are similar to theinitial list manipulation primitives used in the string editingdomain: as both domains can be treated as manipulatingstructured arrays, we are interested in learning differenti-ated, domain-specific abstractions based on a very similarbase language. L0 also includes primitives for queryingattributes of objects on the domain (these are typed gettersthat simply query the object dictionary of attributes) and sev-eral domain-specific functions necessary for manipulatingthese attribute. We deliberately use a much more base levelprogramming language than the high-level, domain-specificlanguage hand-designed in (Johnson et al., 2017); our goalis to learn the necessary abstractions.

We give a semantic gloss for primitives that are not standardLISP primitives.

• if (bool → t → t → t)

• cons (object → list(object) →list(object))

• car (list(object) → object)

• map ((t0 → t1)→ list(t0)→ list(t1))

• fold ((list(t) → list(t)) → (t → list(t) →list(t))→ list(t))

• len (list(t) → int)

• > (list(t) → bool)

• < (list(t) → bool)

• set union (list(t) → list(t) →list(t))

• set intersect (list(t) → list(t) →list(t))

• set difference (list(t) → list(t)→ list(t))

• relate (object → relation →list(t)) Returns an array of objects that sat-isfy a spatial relation with respect to an inputobject.


We also include equality comparators for each of theattribute types (e.g. eq color?; getters for each at-tribute, and setters for each attribute. We also includeinteger constants 0-9 for counting and constants forthe attributes (blue, red, big, small, rubber,metal) based on the original object and spatial relationconstants (Johnson et al., 2017).Domain hyperparameters: We run a coarse hyperparam-eter search based on the baseline model to set the domainhyperparameters. We train all models, including baselinesand ablations, using an enumerative search budget of 1000sper task and run the models for a maximum of 5 iterations.we run multiple random seed replications reordering thetraining set, in the same way as the compositional graphicsdomain. The results in Table 1 also compare a curriculumordering of the training set based on the number of tokensin the synthetic language captions (split on spaces.)

The encoder E(t) is a variant of the RNN-based domain-specific encoder used for text and list editing problems in(Ellis et al., 2021) (as well as the string editing domain). Themodel is trained for a fixed gradient step budget (10,000gradient steps) and we sample equally at random betweensupervision on the solved training tasks (and their solutionprograms in the current DSL) and samples from the jointgenerative model. As with (Ellis et al., 2021), when gen-erating tasks from the generative model, we use randomlysample inputs (on which we execute generated programsto produce an output.) We encode the symbolic scene datastructures with the RNN by encoding a flattened version ofthe scene graph. The scene graph is originally stored as adictionary of attributes; when flattened, we indicate the dic-tionary structure using special tokens to denote the keys andthe start and end of any array delimiters (the original scenegraph is fully reconstructable from the flattened version.)

S 6.2 Results and Additional Qualitative Results

In this section, we discuss additional qualitative results froman in depth exploration of the graphics domain that wereomitted from the main paper for space, but provide addi-tional insight on the behavior of the learned model in thehardest learning domain (based on the differential betweenbaseline and LAPS-augmented performance.)

Learned abstractions and synthesized programs. Fig-ure S4 (supplement) show sample abstractions in the finallibraries Lf for the best performing models in the graph-ics domain as a concrete exemplar of abstractions that arelearned and how they are used, along with sample taskssolved with these abstractions. The figures are shown asdependency graphs to indicate how progressively more com-plex abstractions build on abstractions at prior iterationsof learning; we also show selected probabilities from thetranslation model (depicted are examples from the top-3

primitive translations for a given word; some primitives arenot high probability translations for any word.)

Joint generative model samples. Figure S3 (supplement)shows samples from the joint generative model on the graph-ics domain (programs from the library which are executed toproduce the task example image, and translated to producelanguage annotations) at early and later stages of training,indicating that the joint model itself improves as learningimproves, which itself allows better training for the condi-tional inference model and better abstraction guiding basedon language.

ReferencesAbelson, H. and DiSessa, A. A. Turtle geometry: The

computer as a medium for exploring mathematics. MITpress, 1986.

Andreas, J., Vlachos, A., and Clark, S. Semantic parsing asmachine translation. In Proceedings of the 51st AnnualMeeting of the Association for Computational Linguistics(Volume 2: Short Papers), pp. 47–52, 2013.

Andreas, J., Klein, D., and Levine, S. Learning with latentlanguage. arXiv preprint arXiv:1711.00482, 2017.

Artzi, Y., Das, D., and Petrov, S. Learning compact lexiconsfor ccg semantic parsing. 2014.

Balog, M., Gaunt, A. L., Brockschmidt, M., Nowozin, S.,and Tarlow, D. Deepcoder: Learning to write programs.arXiv preprint arXiv:1611.01989, 2016.

Cho, K., Van Merrienboer, B., Gulcehre, C., Bahdanau,D., Bougares, F., Schwenk, H., and Bengio, Y. Learn-ing phrase representations using rnn encoder-decoderfor statistical machine translation. arXiv preprintarXiv:1406.1078, 2014.

Dechter, E., Malmaud, J., Adams, R. P., and Tenenbaum,J. B. Bootstrap learning via modular concept discovery. InTwenty-Third International Joint Conference on ArtificialIntelligence, 2013.

Devlin, J., Uesato, J., Bhupatiraju, S., Singh, R., Mohamed,A.-r., and Kohli, P. Robustfill: Neural program learningunder noisy i/o. In Proceedings of the 34th InternationalConference on Machine Learning-Volume 70, pp. 990–998. JMLR. org, 2017.

Dumancic, S. and Cropper, A. Inventing abstractions byrefactoring knowledge.

Ellis, K., Morales, L., Sable-Meyer, M., Solar-Lezama, A.,and Tenenbaum, J. Learning libraries of subroutinesfor neurally–guided bayesian program induction. In Ad-vances in Neural Information Processing Systems, pp.7805–7815, 2018.


⍴"

L, 𝜃

𝑑&

��

T

Execute to produce examples

Sample programs from prior

Translate to produce language

Joint generative model fromlearned translation model T

a small square connected by a short line and a smalltriangle as arms

Joint model samples : Iteration 3

a small five gonsa small five gon gona small five gon and

a small five gona small six gona medium five gon

Joint model samples : Iteration 15

small 5 gons in a row

a small 6 gonas arms

a small squareshort line

3 small 5 gonand a small triangle(f41 (λ (x)

((f48 x)) (fn21 1 5 5 x))

(f34 9 6 x)(f17 (λ (x) (get/set (λ (z) (f12 1 (f24 (x z))))(f20 (λ (u) (f15 5 u)) 1 3 3))) v)

((f10 3 x) f34 (f31 (λ (x) x) (λ (y z) z)) 4 9 u))

�� =

𝑑& =

𝜌" = (f6 4 4 1 x)

(f0 5 0 1 x) (f1 1 (f2 5 x))

Figure 3. (left) Joint generative model J over programs sampled from the DSL prior and natural language produced by the translationmodel T (D|L), inferred from solved training tasks. Samples from the model are used to train a neural synthesizer to guide search onmore challenging, unsolved tasks. (right) Samples from the J generative model in the graphics domain shows how program complexityincreases and generated language improves across iterations, as the system both adds richer abstractions to the DSL and learns betteralignments over the solution set, enabling the trained neural model to solve more complex tasks

.

1

0.91 | three0.98 | triangle

. . .

for

move

pen-up

0.94 | four0.89 | square

Original DSL primitives

Learned translationprobabilities p(π | u)

0.31 | line0.31 | short0.09 | a

3

4

2

New primitives added through abstraction learning

f0=(λ (x y z) (for x (λ (u v) (move z y v))))

move pen in parameterized loop f9=(f0 ∞ ε)

0.07 | semicircle

a small semicircle(f19 (f9 0 x))

a medium semicircle(f3 (f9 0 x))

a big semicircle(f9 (* (/ ε 1) 5) x)

f14=(λ (x y) (for 7 (λ (z u) (f9 x u)) y))

a big circle(f14 (logo_DIVL 1 4) x)

a small circle(f14 (logo_DIVL ε 1) x)

two nested circles(f14 ε (f14 ε (f16 x)))

0.16 | circle0.08 | turns0.09 | nested

f4=(λ (x y z) (f0 x (/ 2π y) 1 z))

0.09 | small rotates and draws a unit line

f5=(λ (x y) (f4 x x y))

0.27 | gon0.22 | small

rotational symmetry by number of sides

a small five gon(f5 5 x)

a small nine gon(f5 9 x)

a medium seven gon(f5 2 (f20 7 x))

f6=(λ (x y z u) (for y (λ (v w) (f5 z (f5 x w))) u))

four small squares in a row(f5 2 (f6 1 4 4 x))

six small five gons in a row(f6 1 6 5 x)

... f24=(λ (x y) (f23 (λ (z u) (f21 y 0 x u))))

0.09 | snowflake0.09 | arms

eight sided snowflake with a small seven gon as arms(f24 7 8 x)

five sided snowflake with a short line and a medium five gon as arms(f24 5 (λ (x) (get/set (λ (y) (f2 1 (f41 5 y))) x)) z)

f32=(λ (x) (for x (λ (y z) (move 1 (/ 2π 4) (move 1 (- 2π (/ 2π 4)) z)))))

1.0 | stepped0.64 | staircase0.36 | zigzag

a seven stepped staircase(f32 7 (get/set (λ (x) x) y))

a four stepped staircase (f32 4 (get/set (λ (x) x) y))

a five stepped zigzag(f25 (λ (x) x) 3 8 (f32 5 y)

...

...f17=(λ (x) (pen-up (λ (y) (f16 x y))))

0.67 | separated0.15 | next0.06 | space

a small circle next to a small six gon(f14 ε (f14 ε (f17 2 (f5 6 x))))

a small nine gon next to a medium square(f5 9 (f5 1 (f17 1 (f20 4 x))))

Figure 4. Abstractions and programs learned for the graphics domain. Sample abstractions (right) learned from a minimal starting DSL(left) for solving progressively more complex graphics program synthesis tasks with language annotations. Also shown with translationprobabilities. Our iterative algorithm learns alignment-based translation probabilities between natural language words and programprimitives to guide program search and abstraction (depicted are examples from the top-3 primitive translations for a given word; someprimitives are not high probability translations for any word.


Ellis, K., Wong, C., Nye, M., Sable-Meyer, M., Cary, L.,Morales, L., Hewitt, L., Solar-Lezama, A., and Tenen-baum, J. Dreamcoder: Bootstrapping inductive program-synthesis with wake-sleep library learning. PLDI 2021,2021.

Frank, M. C., Goodman, N. D., and Tenenbaum, J. B. Us-ing speakers’ referential intentions to model early cross-situational word learning. Psychological science, 20(5):578–585, 2009.

Gal, Y. and Blunsom, P. A systematic bayesian treatmentof the ibm alignment models. In Proceedings of the 2013Conference of the North American Chapter of the Associ-ation for Computational Linguistics: Human LanguageTechnologies, pp. 969–977, 2013.

Gandhi, K. and Lake, B. M. Mutual exclusivity as achallenge for deep neural networks. arXiv preprintarXiv:1906.10197, 2019.

Heafield, K. Kenlm: Faster and smaller language modelqueries. In Proceedings of the sixth workshop on statis-tical machine translation, pp. 187–197. Association forComputational Linguistics, 2011.

Johnson, J., Hariharan, B., Van Der Maaten, L., Hoffman,J., Fei-Fei, L., Lawrence Zitnick, C., and Girshick, R.Inferring and executing programs for visual reasoning.In Proceedings of the IEEE International Conference onComputer Vision, pp. 2989–2998, 2017.

Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Fed-erico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C.,Zens, R., et al. Moses: Open source toolkit for statisticalmachine translation. In Proceedings of the 45th annualmeeting of the association for computational linguisticscompanion volume proceedings of the demo and postersessions, pp. 177–180, 2007.

Lake, B. M., Salakhutdinov, R. R., and Tenenbaum, J. One-shot learning by inverting a compositional causal process.In Advances in neural information processing systems,pp. 2526–2534, 2013.

Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B.Human-level concept learning through probabilistic pro-gram induction. Science, 350(6266):1332–1338, 2015.

Liang, P., Jordan, M. I., and Klein, D. Learning programs:A hierarchical bayesian approach. In Proceedings ofthe 27th International Conference on Machine Learning(ICML-10), pp. 639–646, 2010.

Markman, E. M. and Wachtel, G. F. Children’s use of mutualexclusivity to constrain the meanings of words. Cognitivepsychology, 20(2):121–157, 1988.

Nye, M., Hewitt, L., Tenenbaum, J., and Solar-Lezama,A. Learning to infer program sketches. arXiv preprintarXiv:1902.06349, 2019.

Parisotto, E., Mohamed, A.-r., Singh, R., Li, L., Zhou, D.,and Kohli, P. Neuro-symbolic program synthesis. arXivpreprint arXiv:1611.01855, 2016.

Pierce, B. C. Types and programming languages. MITPress, 2002. ISBN 978-0-262-16209-8.

Polosukhin, I. and Skidanov, A. Neural program search:Solving data processing tasks from description and exam-ples. 2018.

Shin, E. C., Allamanis, M., Brockschmidt, M., and Polo-zov, A. Program synthesis and semantic parsing withlearned code idioms. In Advances in Neural InformationProcessing Systems, pp. 10824–10834, 2019.

Wang, Y., Berant, J., and Liang, P. Building a semanticparser overnight. In Proceedings of the 53rd AnnualMeeting of the Association for Computational Linguisticsand the 7th International Joint Conference on NaturalLanguage Processing (Volume 1: Long Papers), pp. 1332–1342, 2015.

Wong, Y. W. and Mooney, R. J. Learning for semantic pars-ing with statistical machine translation. In Proceedingsof the main conference on Human Language Technol-ogy Conference of the North American Chapter of theAssociation of Computational Linguistics, pp. 439–446.Association for Computational Linguistics, 2006.

Yi, K., Wu, J., Gan, C., Torralba, A., Kohli, P., and Tenen-baum, J. Neural-symbolic vqa: Disentangling reasoningfrom vision and language understanding. In Advances inNeural Information Processing Systems, pp. 1031–1042,2018.

Zhang, Y., Pasupat, P., and Liang, P. Macro grammars andholistic triggering for efficient semantic parsing. arXivpreprint arXiv:1707.07806, 2017.

arXiv:2106.11053v3 [cs.LG] 3 May 2022

Documents

Transcript of arXiv:2106.11053v3 [cs.LG] 3 May 2022