|
|
--- |
|
|
language: |
|
|
- en |
|
|
license: apache-2.0 |
|
|
datasets: |
|
|
- teknium/OpenHermes-2.5 |
|
|
base_model: mistral-community/Mixtral-8x22B-v0.1 |
|
|
--- |
|
|
|
|
|
# Model Card for Fireworks Mixtral 8x22b Instruct OH |
|
|
|
|
|
Fireworks Mixtral 8x22b Instruct OH is an instruct version of the latest MoE model from [mistralai](https://huggingface.co/mistralai) - [8x22b](https://huggingface.co/mistral-community/Mixtral-8x22B-v0.1). This |
|
|
model was finetuned on ~10K entries from [OpenHermes](https://huggingface.co/datasets/teknium/OpenHermes-2.5) dataset by [NousResearch](https://huggingface.co/NousResearch). Huge |
|
|
shoutout to [Teknium](https://huggingface.co/teknium) and the NousResearch team for this high-quality SFT dataset. |
|
|
|
|
|
|
|
|
## How to Get Started with the Model |
|
|
|
|
|
Visit our [Playground](https://fireworks.ai/models/fireworks/mixtral-8x22b-instruct-preview) to try the model on a serverless platform, or access it via [API](https://readme.fireworks.ai/reference/createchatcompletion). Fine-tune it using our [guide](https://readme.fireworks.ai/docs/fine-tuning-models). |
|
|
|
|
|
The model uses Vicuna chat template: |
|
|
``` |
|
|
USER: ... |
|
|
|
|
|
ASSISTANT: ... |
|
|
``` |
|
|
or: |
|
|
``` |
|
|
SYSTEM: ... |
|
|
|
|
|
USER: ... |
|
|
|
|
|
ASSISTANT: ... |
|
|
``` |
|
|
|
|
|
### Training Data |
|
|
|
|
|
To save training time, we sample & format 10K samples from [OpenHermes 2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5) dataset. We perform reformatting so as to |
|
|
suit the vicuna chat template. Apart from that dataset is used as is. |
|
|
|
|
|
|
|
|
#### Training Setup |
|
|
|
|
|
The model was trained on 8xA100 [80G] using Deepspeed Zero 3. We had to fix an existing bug with deepspeed zero 3 where interaction between zero 3 & MoE causes |
|
|
training to get stuck. |
|
|
|
|
|
|
|
|
## Model Examination |
|
|
|
|
|
<details Sample 1> |
|
|
<summary>Itinerary Question</summary> |
|
|
<code> |
|
|
SYSTEM: You are a helpful assistant. Please be kind, patient and truthful in answering the questions. |
|
|
|
|
|
USER: Create a travel itinerary for the bahamas. Spare no detail. |
|
|
|
|
|
ASSISTANT: Sure, I'd be happy to help you plan a trip to the Bahamas! Here's a detailed itinerary for your consideration: |
|
|
|
|
|
Day 1: Arrival in Nassau |
|
|
- Arrive at Lynden Pindling International Airport in Nassau, the capital city of the Bahamas. |
|
|
- Check into your hotel or resort. |
|
|
- Spend the afternoon exploring the city, visiting places like the Queen's Staircase, Fort Fincastle, and the Straw Market. |
|
|
- Enjoy a delicious Bahamian dinner at one of the local restaurants. |
|
|
|
|
|
Day 2: Nassau City Tour and Atlantis Resort |
|
|
- Start your day with a guided city tour of Nassau, visiting historical sites like the Government House, Parliament Square, and the Pirates of Nassau Museum. |
|
|
- In the afternoon, head to the famous Atlantis Resort on Paradise Island. Explore the resort's attractions, such as the Aquaventure Water Park, the Marine Habitat, and the casino. |
|
|
- End the day with a relaxing dinner at one of the resort's restaurants. |
|
|
|
|
|
Day 3: Exuma Cays Day Trip |
|
|
- Take a day trip to the Exuma Cays, a group of over 365 islands and cays. |
|
|
- Visit popular spots like the Swimming Pigs at Big Major Cay, the Thunderball Grotto (featured in the James Bond movie "Thunderball"), and the stunning sandbars. |
|
|
- Enjoy a picnic lunch on one of the pristine beaches. |
|
|
- Return to Nassau in the evening. |
|
|
|
|
|
Day 4: Blue Lagoon Island and Beach Day |
|
|
- Spend the morning at Blue Lagoon Island, a private island just a short boat ride from Nassau. |
|
|
- Enjoy activities like swimming, snorkeling, kayaking, or simply relaxing on the beach. |
|
|
- In the afternoon, head back to Nassau and spend some time at one of the city's beautiful beaches, such as Cable Beach or Junkanoo Beach. |
|
|
- Have dinner at a local seafood restaurant. |
|
|
|
|
|
Day 5: Departure |
|
|
- Depending on your flight schedule, you may have some time to explore Nassau further or do some last-minute shopping before heading to the airport for your departure. |
|
|
|
|
|
I hope this itinerary helps you plan your trip to the Bahamas! Let me know if you have any other questions or need further assistance. |
|
|
</code> |
|
|
</details> |
|
|
|
|
|
<details Sample 2> |
|
|
<summary>QnA Question</summary> |
|
|
<code> |
|
|
SYSTEM: You are a helpful assistant. Please be kind, patient and truthful in answering the questions. |
|
|
|
|
|
USER: What is the medical term for strong headache on one side of the head? |
|
|
|
|
|
ASSISTANT: The medical term for a strong headache on one side of the head is called a migraine. |
|
|
</code> |
|
|
</details> |
|
|
|
|
|
<details Sample 3> |
|
|
<summary>Knowledge Question</summary> |
|
|
<code> |
|
|
SYSTEM: You are a helpful assistant. Please be kind, patient and truthful in answering the questions. |
|
|
|
|
|
USER: How can I get stock price of stripe? |
|
|
|
|
|
ASSISTANT: Stripe is a private company and does not have publicly traded stock. Therefore, you cannot get the stock price for Stripe. |
|
|
</code> |
|
|
</details> |
|
|
|
