31

GitHub - cloudfetch/cloudquery: Turn any website to serverless API (support SPA!...

 5 years ago
source link: https://github.com/cloudfetch/cloudquery
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.

README.md

CloudQuery

Turn any website to serverless API (with SPA support!)

Try online: https://2uvlp0y13b.execute-api.us-west-2.amazonaws.com/staging/

API

Example url: https://2uvlp0y13b.execute-api.us-west-2.amazonaws.com/staging/query?url=https://news.ycombinator.com/best&selectors=*:nth-child(1) > * > *:nth-child(2) > *:nth-child(2) > :nth-child(1),:nth-child(3) *:nth-child(1) > *:nth-child(3) > *:nth-child(1)

Query params:

  • url: url of the webpage you want to fetch
  • selectors: css selectors describing the elements interestes you

Returns:

  • innerText: element innerText
  • href: element href

Run locally

npm i
npm start

How to deploy your own CloudQuery to AWS lambda

1. Configuration

  • update profile(AWS Credential Profiles) in up.json to use your own aws account
  • update rateLimit in config.json to set your own rate limit (Default: 5 request/hour)

2. Deploy

  1. npm install
  2. install up
  3. Input up in terminal and hit enter :)
  4. up url to see the API baseURL of your CloudQuery

Contribute

Backend

vim app.js

Frontend

cd frontend
# develop
npm start
# build
npm run build

Thanks

  • up for deploying serverless API to AWS with ease
  • serverless-chrome for running chrome on AWS lambda
  • finder for making it easy to select elements on webpage

report

Recommend

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK