arXiv subjects

Mark Johnson

Publications and source records attributed to Mark Johnson.

At least 19 recordsLinked to original sources

arXiv2025

Modelling Child Learning and Parsing of Long-range Syntactic Dependencies

This work develops a probabilistic child language acquisition model to learn a range of linguistic phenonmena, most notably long-range syntactic dependencies of the sort found in object wh-questions, among other constructions. The model is trained on a corpus of real child-directed speech, where each utterance is paired with a logical form as a meaning representation. It then learns both word meanings and language-specific syntax simultaneously. After training, the model can deduce the correct parse tree and word meanings for a given utterance-meaning pair, and can infer the meaning if given only the utterance. The successful modelling of long-range dependencies is theoretically important because it exploits aspects of the model that are, in general, trans-context-free.

cs.CL↗

arXiv2024

Mastering the Craft of Data Synthesis for CodeLLMs

Large language models (LLMs) have shown impressive performance in \emph{code} understanding and generation, making coding tasks a key focus for researchers due to their practical applications and value as a testbed for LLM evaluation. Data synthesis and filtering techniques have been widely adopted and shown to be highly effective in this context. In this paper, we present a focused survey and taxonomy of these techniques, emphasizing recent advancements. We highlight key challenges, explore future research directions, and offer practical guidance for new researchers entering the field.

cs.SE↗

arXiv2024

A Language-agnostic Model of Child Language Acquisition

This work reimplements a recent semantic bootstrapping child-language acquisition model, which was originally designed for English, and trains it to learn a new language: Hebrew. The model learns from pairs of utterances and logical forms as meaning representations, and acquires both syntax and word meanings simultaneously. The results show that the model mostly transfers to Hebrew, but that a number of factors, including the richer morphology in Hebrew, makes the learning slower and less robust. This suggests that a clear direction for future work is to enable the model to leverage the similarities between different word forms.

cs.CL↗

arXiv2023

Sources of Hallucination by Large Language Models on Inference Tasks

Large Language Models (LLMs) are claimed to be capable of Natural Language Inference (NLI), necessary for applied tasks like question answering and summarization. We present a series of behavioral studies on several LLM families (LLaMA, GPT-3.5, and PaLM) which probe their behavior using controlled experiments. We establish two biases originating from pretraining which predict much of their behavior, and show that these are major sources of hallucination in generative LLMs. First, memorization at the level of sentences: we show that, regardless of the premise, models falsely label NLI test samples as entailing when the hypothesis is attested in training data, and that entities are used as ``indices'' to access the memorized data. Second, statistical patterns of usage learned at the level of corpora: we further show a similar effect when the premise predicate is less frequent than that of the hypothesis in the training data, a bias following from previous studies. We demonstrate that LLMs perform significantly worse on NLI test samples which do not conform to these biases than those which do, and we offer these as valuable controls for future LLM evaluation.

cs.CL↗

arXiv2022

A dataset of direct observations of sea ice drift and waves in ice

Variability in sea ice conditions, combined with strong couplings to the atmosphere and the ocean, lead to a broad range of complex sea ice dynamics. More in-situ measurements are needed to better identify the phenomena and mechanisms that govern sea ice growth, drift, and breakup. To this end, we have gathered a dataset of in-situ observations of sea ice drift and waves in ice. A total of 15 deployments were performed over a period of 5 years in both the Arctic and Antarctic, involving 72 instruments. These provide both GPS drift tracks, and measurements of waves in ice. The data can, in turn, be used for tuning sea ice drift models, investigating waves damping by sea ice, and helping calibrate other sea ice measurement techniques, such as satellite based observations.

physics.geo-ph↗

arXiv2022

Smoothing Entailment Graphs with Language Models

The diversity and Zipfian frequency distribution of natural language predicates in corpora leads to sparsity in Entailment Graphs (EGs) built by Open Relation Extraction (ORE). EGs are computationally efficient and explainable models of natural language inference, but as symbolic models, they fail if a novel premise or hypothesis vertex is missing at test-time. We present theory and methodology for overcoming such sparsity in symbolic models. First, we introduce a theory of optimal smoothing of EGs by constructing transitive chains. We then demonstrate an efficient, open-domain, and unsupervised smoothing method using an off-the-shelf Language Model to find approximations of missing premise predicates. This improves recall by 25.1 and 16.3 percentage points on two difficult directional entailment datasets, while raising average precision and maintaining model explainability. Further, in a QA task we show that EG smoothing is most useful for answering questions with lesser supporting text, where missing premise predicates are more costly. Finally, controlled experiments with WordNet confirm our theory and show that hypothesis smoothing is difficult, but possible in principle.

cs.CL↗

arXiv2022

OpenMetBuoy-V2021: an easy-to-build, affordable, customizable, open source instrument for oceanographic measurements of drift and waves in sea ice and the open ocean

There is a wide consensus within the polar science, meteorology, and oceanography communities that more in-situ observations of the ocean, atmosphere, and sea ice, are required to further improve operational forecasting model skills. Traditionally, the volume of such measurements has been limited by the high cost of commercially available instruments. An increasingly attractive solution to this cost issue is to use instruments produced in-house from open source hardware, firmware, and post processing building blocks. In the present work, we release the next iteration of the open source drifter and waves monitoring instruments. The new design is both significantly less expensive, much easier to build and assemble for people without specific microelectronics and programming competence, more easily extendable and customizable, and two orders of magnitude more power efficient. Improving performance and reducing noise levels and costs compared with our previous generation of instruments is possible in large part thanks to progress from the electronics component industry. As a result, we believe that this will allow scientists in geosciences to increase by an order of magnitude the amount of in-situ data they can collect under a constant instrumentation budget. In the following, we offer 1) detailed overview of our hardware and software solution, 2) in-situ validation and benchmarking of our instrument, 3) full open source release of both hardware and software blueprints. We hope that this work, and the associated open source release, may be a milestone that will allow our scientific fields to transition towards open source, community driven instrumentation. We believe that this could have a considerable impact on many fields, by making in-situ instrumentation at least an order of magnitude less expensive and more customizable than it has been for the last 50 years.

physics.ins-det↗

arXiv2021

Three-dimensional Cooperative Localization of Commercial-Off-The-Shelf Sensors

Many location-based services use Received Signal Strength (RSS) measurements due to their universal availability. In this paper, we study the association of a large number of low-cost Internet-of-Things (IoT) sensors and their possible installation locations, which can enable various sensing and automation-related applications. We propose an efficient approach to solve the corresponding permutation combinatorial optimization problem, which integrates continuous space cooperative localization and permutation space likelihood ascent search. A convex relaxation-based optimization is designed to estimate the coarse locations of blindfolded devices in continuous 3D spaces, which are then projected to the feasible permutation space. An efficient Cram\'er-Rao Lower Bound based likelihood ascent search algorithm is proposed to refine the solution. Extensive experiments were conducted to evaluate the performance of the proposed approach, which show that the proposed approach significantly outperforms state-of-the-art combinatorial optimization algorithms and achieves close-to-100% accuracy with affordable execution time.

cs.IR↗

arXiv2021

Blindness to Modality Helps Entailment Graph Mining

Understanding linguistic modality is widely seen as important for downstream tasks such as Question Answering and Knowledge Graph Population. Entailment Graph learning might also be expected to benefit from attention to modality. We build Entailment Graphs using a news corpus filtered with a modality parser, and show that stripping modal modifiers from predicates in fact increases performance. This suggests that for some tasks, the pragmatics of modal modification of predicates allows them to contribute as evidence of entailment.

cs.CL↗

arXiv2021

Incorporating Temporal Information in Entailment Graph Mining

We present a novel method for injecting temporality into entailment graphs to address the problem of spurious entailments, which may arise from similar but temporally distinct events involving the same pair of entities. We focus on the sports domain in which the same pairs of teams play on different occasions, with different outcomes. We present an unsupervised model that aims to learn entailments such as win/lose $\rightarrow$ play, while avoiding the pitfall of learning non-entailments such as win $\not\rightarrow$ lose. We evaluate our model on a manually constructed dataset, showing that incorporating time intervals and applying a temporal window around them, are effective strategies.

cs.CL↗

arXiv2021

Neural Rule-Execution Tracking Machine For Transformer-Based Text Generation

Sequence-to-Sequence (S2S) neural text generation models, especially the pre-trained ones (e.g., BART and T5), have exhibited compelling performance on various natural language generation tasks. However, the black-box nature of these models limits their application in tasks where specific rules (e.g., controllable constraints, prior knowledge) need to be executed. Previous works either design specific model structure (e.g., Copy Mechanism corresponding to the rule "the generated output should include certain words in the source input") or implement specialized inference algorithm (e.g., Constrained Beam Search) to execute particular rules through the text generation. These methods require careful design case-by-case and are difficult to support multiple rules concurrently. In this paper, we propose a novel module named Neural Rule-Execution Tracking Machine that can be equipped into various transformer-based generators to leverage multiple rules simultaneously to guide the neural generation model for superior generation performance in a unified and scalable way. Extensive experimental results on several benchmarks verify the effectiveness of our proposed model in both controllable and general text generation.

cs.CL↗

arXiv2021

Multivalent Entailment Graphs for Question Answering

Drawing inferences between open-domain natural language predicates is a necessity for true language understanding. There has been much progress in unsupervised learning of entailment graphs for this purpose. We make three contributions: (1) we reinterpret the Distributional Inclusion Hypothesis to model entailment between predicates of different valencies, like DEFEAT(Biden, Trump) entails WIN(Biden); (2) we actualize this theory by learning unsupervised Multivalent Entailment Graphs of open-domain predicates; and (3) we demonstrate the capabilities of these graphs on a novel question answering task. We show that directional entailment is more helpful for inference than bidirectional similarity on questions of fine-grained semantics. We also show that drawing on evidence across valencies answers more questions than by using only the same valency evidence.

cs.CL↗

arXiv2021

There and Back Again: Self-supervised Multispectral Correspondence Estimation

Across a wide range of applications, from autonomous vehicles to medical imaging, multi-spectral images provide an opportunity to extract additional information not present in color images. One of the most important steps in making this information readily available is the accurate estimation of dense correspondences between different spectra. Due to the nature of cross-spectral images, most correspondence solving techniques for the visual domain are simply not applicable. Furthermore, most cross-spectral techniques utilize spectra-specific characteristics to perform the alignment. In this work, we aim to address the dense correspondence estimation problem in a way that generalizes to more than one spectrum. We do this by introducing a novel cycle-consistency metric that allows us to self-supervise. This, combined with our spectra-agnostic loss functions, allows us to train the same network across multiple spectra. We demonstrate our approach on the challenging task of dense RGB-FIR correspondence estimation. We also show the performance of our unmodified network on the cases of RGB-NIR and RGB-RGB, where we achieve higher accuracy than similar self-supervised approaches. Our work shows that cross-spectral correspondence estimation can be solved in a common framework that learns to generalize alignment across spectra.

cs.CV↗

arXiv2021

ECOL-R: Encouraging Copying in Novel Object Captioning with Reinforcement Learning

Novel Object Captioning is a zero-shot Image Captioning task requiring describing objects not seen in the training captions, but for which information is available from external object detectors. The key challenge is to select and describe all salient detected novel objects in the input images. In this paper, we focus on this challenge and propose the ECOL-R model (Encouraging Copying of Object Labels with Reinforced Learning), a copy-augmented transformer model that is encouraged to accurately describe the novel object labels. This is achieved via a specialised reward function in the SCST reinforcement learning framework (Rennie et al., 2017) that encourages novel object mentions while maintaining the caption quality. We further restrict the SCST training to the images where detected objects are mentioned in reference captions to train the ECOL-R model. We additionally improve our copy mechanism via Abstract Labels, which transfer knowledge from known to novel object types, and a Morphological Selector, which determines the appropriate inflected forms of novel object labels. The resulting model sets new state-of-the-art on the nocaps (Agrawal et al., 2019) and held-out COCO (Hendricks et al., 2016) benchmarks.

cs.CV↗

arXiv2020

Detecting and Exorcising Statistical Demons from Language Models with Anti-Models of Negative Data

It's been said that "Language Models are Unsupervised Multitask Learners." Indeed, self-supervised language models trained on "positive" examples of English text generalize in desirable ways to many natural language tasks. But if such models can stray so far from an initial self-supervision objective, a wayward model might generalize in undesirable ways too, say to nonsensical "negative" examples of unnatural language. A key question in this work is: do language models trained on (positive) training data also generalize to (negative) test data? We use this question as a contrivance to assess the extent to which language models learn undesirable properties of text, such as n-grams, that might interfere with the learning of more desirable properties of text, such as syntax. We find that within a model family, as the number of parameters, training epochs, and data set size increase, so does a model's ability to generalize to negative n-gram data, indicating standard self-supervision generalizes too far. We propose a form of inductive bias that attenuates such undesirable signals with negative data distributions automatically learned from positive data. We apply the method to remove n-gram signals from LSTMs and find that doing so causes them to favor syntactic signals, as demonstrated by large error reductions (up to 46% on the hardest cases) on a syntactic subject-verb agreement task.

cs.CL↗

arXiv2020

End-to-End Speech Recognition and Disfluency Removal

Disfluency detection is usually an intermediate step between an automatic speech recognition (ASR) system and a downstream task. By contrast, this paper aims to investigate the task of end-to-end speech recognition and disfluency removal. We specifically explore whether it is possible to train an ASR model to directly map disfluent speech into fluent transcripts, without relying on a separate disfluency detection model. We show that end-to-end models do learn to directly generate fluent transcripts; however, their performance is slightly worse than a baseline pipeline approach consisting of an ASR system and a disfluency detection model. We also propose two new metrics that can be used for evaluating integrated ASR and disfluency models. The findings of this paper can serve as a benchmark for further research on the task of end-to-end speech recognition and disfluency removal in the future.

eess.AS↗

arXiv2020

Low-Complexity System and Algorithm for an Emergency Ventilator Sensor and Alarm

In response to the shortage of ventilators caused by the COVID-19 pandemic, many organizations have designed low-cost emergency ventilators. Many of these devices are pressure-cycled pneumatic ventilators, which are easy to produce but often do not include the sensing or alarm features found on commercial ventilators. This work reports a low-cost, easy-to-produce electronic sensor and alarm system for pressure-cycled ventilators that estimates clinically useful metrics such as pressure and respiratory rate and sounds an alarm when the ventilator malfunctions. A low-complexity signal processing algorithm uses a pair of nonlinear recursive envelope trackers to monitor the signal from an electronic pressure sensor connected to the patient airway. The algorithm, inspired by those used in hearing aids, requires little memory and performs only a few calculations on each sample so that it can run on nearly any microcontroller.

eess.SP↗

arXiv2020

Improving Disfluency Detection by Self-Training a Self-Attentive Model

Self-attentive neural syntactic parsers using contextualized word embeddings (e.g. ELMo or BERT) currently produce state-of-the-art results in joint parsing and disfluency detection in speech transcripts. Since the contextualized word embeddings are pre-trained on a large amount of unlabeled data, using additional unlabeled data to train a neural model might seem redundant. However, we show that self-training - a semi-supervised technique for incorporating unlabeled data - sets a new state-of-the-art for the self-attentive parser on disfluency detection, demonstrating that self-training provides benefits orthogonal to the pre-trained contextualized word representations. We also show that ensembling self-trained parsers provides further gains for disfluency detection.

cs.CL↗