Community
Search Options
Search Options
Log in
Skip to main content (Press Enter).
Sign in
Skip auxiliary navigation (Press Enter).
AI and Data Science
Topic areas
AI and DS Skills
Decision Optimization
Embeddable AI
Global AI and Data Science
IBM Advanced Studies
SPSS Statistics
watsonx Assistant
Watson Discovery
User groups
Events
Upcoming AI Events
IBM TechXchange Webinars
All IBM TechXchange Community Events
Participate
Gamification Program
Community Manager's Welcome
Post to Forum
Share a Resource
Share Your Expertise
Blogging on the Community
Connect with Data Science Users
All IBM TechXchange Community Users
Resources
IBM TechXchange Group
AI Learning
IBM Champions
IBM Cloud Support
IBM Documentation
IBM Support
IBM Technology Zone
IBM Training
TechXchange Conference
IBM TechXchange Conference 2024
Marketplace
Marketplace
AI and Data Science
Master the art of data science.
Join now
Skip main navigation (Press Enter).
Toggle navigation
Search Options
Global AI and Data Science
Group Navigator
View Only
Community Home
Discussion
2.2K
Library
267
Blogs
724
Events
9
Members
27.7K
Expand all
|
Collapse all
sort by most recent
sort by thread
RoBERTa: a Robustly Optimized BERT Pretraining Approach
William Roberts
Wed September 04, 2019 12:55 AM
How do you optimize language model pre-training when training tends to be computationally expensive and ...
William Roberts
Mon September 23, 2019 06:31 PM
RoBERTa is surely going to drop out of SOTA, soon! ------------------------------ William Roberts ------------------------------ ...
1.
RoBERTa: a Robustly Optimized BERT Pretraining Approach
0
Like
William Roberts
Posted Wed September 04, 2019 12:55 AM
Edited by System Fri January 20, 2023 04:09 PM
Reply
Options Dropdown
How do you optimize language model pre-training when training tends to be computationally expensive and executed on differing datasets? Maybe
RoBERTa
has the answers. Facebook's pre-training recipe appears to have greatly improved on BERT's bench-marking performance. What do you think is in store for RoBERTa?
Image from:
https://github.com/facebookresearch/LASER/blob/master/tasks/WikiMatrix/WikiMatrix-sizes.pdf
------------------------------
William Roberts
------------------------------
#GlobalAIandDataScience
#GlobalDataScience
2.
RE: RoBERTa: a Robustly Optimized BERT Pretraining Approach
0
Like
William Roberts
Posted Mon September 23, 2019 06:31 PM
Reply
Options Dropdown
RoBERTa is surely going to drop out of SOTA, soon!
------------------------------
William Roberts
------------------------------
Original Message
Original Message:
Sent: Wed September 04, 2019 12:54 AM
From: William Roberts
Subject: RoBERTa: a Robustly Optimized BERT Pretraining Approach
How do you optimize language model pre-training when training tends to be computationally expensive and executed on differing datasets? Maybe
RoBERTa
has the answers. Facebook's pre-training recipe appears to have greatly improved on BERT's bench-marking performance. What do you think is in store for RoBERTa?
Image from:
https://github.com/facebookresearch/LASER/blob/master/tasks/WikiMatrix/WikiMatrix-sizes.pdf
------------------------------
William Roberts
------------------------------
#GlobalAIandDataScience
#GlobalDataScience
×
New Best Answer
This thread already has a best answer. Would you like to mark this message as the new best answer?
IBM Community Home
Browse
Discussions
Resources
Groups
Events
IBM TechXchange Conference 2023
IBM Community Webinars
All IBM Community Events
Participate
Gamification Program
Community Manager's Welcome
Post to Forum
Share a Resource
Blogging on the Community
All IBM Community Users
Resources
Community Front Porch
IBM Champions
IBM Cloud Support
IBM Documentation
IBM Support
IBM Technology Zone
IBM Training
Marketplace
Marketplace
AI and Data Science
Topic areas
AI and DS Skills
Decision Optimization
Embeddable AI
Global AI and Data Science
IBM Advanced Studies
SPSS Statistics
watsonx Assistant
Watson Discovery
User groups
Events
Upcoming AI Events
IBM TechXchange Webinars
All IBM TechXchange Community Events
Participate
Gamification Program
Community Manager's Welcome
Post to Forum
Share a Resource
Share Your Expertise
Blogging on the Community
Connect with Data Science Users
All IBM TechXchange Community Users
Resources
IBM TechXchange Group
AI Learning
IBM Champions
IBM Cloud Support
IBM Documentation
IBM Support
IBM Technology Zone
IBM Training
TechXchange Conference
IBM TechXchange Conference 2024
Marketplace
Marketplace
Copyright © 2019 IBM Data Science Community. All rights reserved.
Powered by Higher Logic