Instructions to use defog/sqlcoder-70b-alpha with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use defog/sqlcoder-70b-alpha with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="defog/sqlcoder-70b-alpha")# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("defog/sqlcoder-70b-alpha") model = AutoModelForCausalLM.from_pretrained("defog/sqlcoder-70b-alpha") - Notebooks
- Google Colab
- Kaggle
- Local Apps
- vLLM
How to use defog/sqlcoder-70b-alpha with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "defog/sqlcoder-70b-alpha" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "defog/sqlcoder-70b-alpha", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/defog/sqlcoder-70b-alpha
- SGLang
How to use defog/sqlcoder-70b-alpha with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "defog/sqlcoder-70b-alpha" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "defog/sqlcoder-70b-alpha", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "defog/sqlcoder-70b-alpha" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "defog/sqlcoder-70b-alpha", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use defog/sqlcoder-70b-alpha with Docker Model Runner:
docker model run hf.co/defog/sqlcoder-70b-alpha
About Supervised fine tuning
If we want to fine tune the model with our database schema, what would be the optimal format for training and number of training samples ?
And could we use methods like LORA here with hugging face interface of PEFT.
not much on the internet about schema based fine tuning this model, guess defog offers that as a paid service.
@rishdotblog @wongjingping could you please provide your thoughts. I'm interested in fine-tuning it.
@Iamexperimenting yes we offer that as a paid service, please feel free to reach out to rishabh@defog.ai for commercial enquiries.