Oldest Papers

Using data from the nationally representative Education Longitudinal Study of 2002, this study examined how journalism participation in high school relates to subsequent academic outcomes. The analysis statistically controlled for a host of correlates of academic achievement, isolating the associations between journalism participation and subsequent outcomes. Results indicated that students who take more journalism in high school score higher than their peers on standardized tests of English; are more likely to major in journalism or related fields; and when they do, have higher grades in college English. Students who participate in extracurricular journalism also see some of these gains.

No ratings yet

View paper →

BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer

Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, Peng Jiang

arXiv·2019

Modeling users' dynamic and evolving preferences from their historical behaviors is challenging and crucial for recommendation systems. Previous methods employ sequential neural networks (e.g., Recurrent Neural Network) to encode users' historical interactions from left to right into hidden representations for making recommendations. Although these methods achieve satisfactory results, they often assume a rigidly ordered sequence which is not always practical. We argue that such left-to-right unidirectional architectures restrict the power of the historical sequence representations. For this purpose, we introduce a Bidirectional Encoder Representations from Transformers for sequential Recommendation (BERT4Rec). However, jointly conditioning on both left and right context in deep bidirectional model would make the training become trivial since each item can indirectly "see the target item". To address this problem, we train the bidirectional model using the Cloze task, predicting the masked items in the sequence by jointly conditioning on their left and right context. Comparing with predicting the next item at each position in a sequence, the Cloze task can produce more samples to train a more powerful bidirectional model. Extensive experiments on four benchmark datasets show that our model outperforms various state-of-the-art sequential models consistently.

No ratings yet

View paper →

PreviousPage 141 of 549Next

Generating Long Sequences with Sparse Transformers

Rewon Child, Scott Gray, Alec Radford, Ilya Sutskever

arXiv·2019

Transformers are powerful sequence models, but require time and memory that grows quadratically with the sequence length. In this paper we introduce sparse factorizations of the attention matrix which reduce this to $O(n \sqrt{n})$. We also introduce a) a variation on architecture and initialization to train deeper networks, b) the recomputation of attention matrices to save memory, and c) fast attention kernels for training. We call networks with these changes Sparse Transformers, and show they can model sequences tens of thousands of timesteps long using hundreds of layers. We use the same architecture to model images, audio, and text from raw bytes, setting a new state of the art for density modeling of Enwik8, CIFAR-10, and ImageNet-64. We generate unconditional samples that demonstrate global coherence and great diversity, and show it is possible in principle to use self-attention to model sequences of length one million or more.

No ratings yet

View paper →

Language Specificity in Phonetic Cue Weighting: Monolingual and Bilingual Perception of the Stop Voicing Contrast in English and Spanish

Jessamyn Schertz, Kathy Carbonell, Andrew J. Lotto

Phonetica·2019·10 citations

Background/Aims: This work examines the perception of the stop voicing contrast in Spanish and English along four acoustic dimensions, comparing mono- lingual and bilingual listeners. Our primary goals are to test the extent to which cue-weighting strategies are language-specific in monolinguals, and whether this language specificity extends to bilingual listeners. Methods: Participants catego- rized sounds varying in voice onset time (VOT, the primary cue to the contrast) and three secondary cues: fundamental frequency at vowel onset, first formant (F1) on- set frequency, and stop closure duration. Listeners heard acoustically identical tar- get stimuli, within language-specific carrier phrases, in English and Spanish modes. Results: While all listener groups used all cues, monolingual English listeners relied more on F1, and less on closure duration, than monolingual Spanish listeners, indi- cating language specificity in cue use. Early bilingual listeners used the three sec- ondary cues similarly in English and Spanish, despite showing language-specific VOT boundaries. Conclusion: While our findings reinforce previous work demon- strating language-specific phonetic representations in bilinguals in terms of VOT boundary, they suggest that this specificity may not extend straightforwardly to cue-weighting strategies.

No ratings yet

View paper →

A Recipe for Training Neural Networks

Andrej Karpathy

karpathy.github.io·2019

Some few weeks ago I posted a tweet on “the most common neural net mistakes”, listing a few common gotchas related to training neural nets. The tweet got quite a bit more engagement than I anticipated (including a webinar :)). Clearly, a lot of people have personally encountered the large gap between “here is how a convolutional layer works” and “our convnet achieves state of the art results”. So I thought it could be fun to brush off my dusty blog to expand my tweet to the long form that this topic deserves. However, instead of going into an enumeration of more common errors or fleshing them out, I wanted to dig a bit deeper and talk about how one can avoid making these errors altogether (or fix them very fast). The trick to doing so is to follow a certain process, which as far as I can tell is not very often documented. Let’s start with two important observations that motivate it.

★ 5.0 (1)

View paper →

Ray Interference: a Source of Plateaus in Deep Reinforcement Learning

Tom Schaul, Diana Borsa, Joseph Modayil, Razvan Pascanu

arXiv·2019

Rather than proposing a new method, this paper investigates an issue present in existing learning algorithms. We study the learning dynamics of reinforcement learning (RL), specifically a characteristic coupling between learning and data generation that arises because RL agents control their future data distribution. In the presence of function approximation, this coupling can lead to a problematic type of 'ray interference', characterized by learning dynamics that sequentially traverse a number of performance plateaus, effectively constraining the agent to learn one thing at a time even when learning in parallel is better. We establish the conditions under which ray interference occurs, show its relation to saddle points and obtain the exact learning dynamics in a restricted setting. We characterize a number of its properties and discuss possible remedies.

No ratings yet

View paper →

Deep-IRT: Make Deep Learning Based Knowledge Tracing Explainable Using Item Response Theory

Chun-Kit Yeung

arXiv·2019

Deep learning based knowledge tracing model has been shown to outperform traditional knowledge tracing model without the need for human-engineered features, yet its parameters and representations have long been criticized for not being explainable. In this paper, we propose Deep-IRT which is a synthesis of the item response theory (IRT) model and a knowledge tracing model that is based on the deep neural network architecture called dynamic key-value memory network (DKVMN) to make deep learning based knowledge tracing explainable. Specifically, we use the DKVMN model to process the student's learning trajectory and estimate the student ability level and the item difficulty level over time. Then, we use the IRT model to estimate the probability that a student will answer an item correctly using the estimated student ability and the item difficulty. Experiments show that the Deep-IRT model retains the performance of the DKVMN model, while it provides a direct psychological interpretation of both students and items.

No ratings yet

View paper →

From Slash Fan Fiction to Crip Fan Fiction: What Role Does Disability Have in Fandom?

Derek Newman-Stille

Canadian Journal of Disability Studies·2019·1 citations

Slash fiction is perceived by scholars like Henry Jenkins as capable of presenting a counterhegemonic message that critically questions and disrupts power structures in the production of fiction. Slash fiction presents a critical queering of characters, disrupting the heterocentrism of canonical fiction. Slash fiction is a creation of fan fiction where canonically heterosexual couples are paired with one another in love relationships, allowing for an imagined queer potential. Even though slash, with its queering of relationships would seem to be a doorway into empowerment for disability fiction - replacing one oppressed identity (queer) for another (disability), many of the conventions of slash, mixed with the overwhelming social power of stereotypes around disability serve to further replicate patterns of oppression upon disabled characters. One of the conventions of slash fiction is the need to make canonically straight male characters more vulnerable, more willing to explore their vulnerability in relationships. This vulnerability allows for male protagonists to disrupt the rigid boundaries of patriarchal, heterosexist constructions of masculinity by making the characters more open to vulnerabilities, which tend to be constructed as threats to the construction of patriarchal masculinity. Because of disability’s cultural association with vulnerability in the cultural imagination, disablement is often utilized by slash fiction authors as a means of achieving vulnerability of the characters in a slash fiction relationship. These relationships are often referred to as “Hurt/Comfort” or “H/C” and often depend on the assumption that disablement represents a weakening of the disabled character, problematically representing disability as weakness. Through an examination of the association between slash and disability on the popular fan fiction site Archive of Our Own, this paper illustrates that although slash fiction has the potential to represent a liberatory counterhegemonic text, it fails to do so where disability is concerned and relies on tropes and assumptions about disability in order to ‘queer’ hegemonic texts.

★ 3.0 (1)

View paper →

Oldest Papers

Scientific Gullibility

‘We Asked 3,601 People’: A Nationwide Public Opinion Poll on Attitudes Towards Archaeology and Archaeological Assets in Turkey

When “Journalism Kids” Do Better: A Reassessment of Secondary and Postsecondary Achievement and Activities

BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer

Logic is a harsh mistress: welfare economics for economists | Journal of Institutional Economics | Cambridge Core

Generating Long Sequences with Sparse Transformers

Language Specificity in Phonetic Cue Weighting: Monolingual and Bilingual Perception of the Stop Voicing Contrast in English and Spanish

A Recipe for Training Neural Networks

Ray Interference: a Source of Plateaus in Deep Reinforcement Learning

Deep-IRT: Make Deep Learning Based Knowledge Tracing Explainable Using Item Response Theory

From Slash Fan Fiction to Crip Fan Fiction: What Role Does Disability Have in Fandom?

“THE STORY OF A DEAD SELF”: