In this final post of the mini-series toward hybrid storage, we are going to set up S3(minio), and Postgres and split the data between these 2 data sources. 

The method InitFixture will accomplish the following:


not really in the cloud, but with the help of docker-compose, here is the method call from init:

func init() {
   // create fake data from a year a go
   to := time.Now()
   from := to.AddDate(-1, 0, 0)

   totalTrades := 10000
   transferToS3 := 8000 // 1 t/m 7999 going to be stored in folder
   fixtures.InitFixture(from, to, totalTrades, transferToS3)
}

After the database is populated, we are going to download the data in total to test.csv in the root folder.


func main() {

   to := time.Now()
   from := to.AddDate(-1, 0, 0)
   from = from.Add(-1 * time.Hour)

   fetchSize := 1000

   source1 := datasource.NewS3Reader(context.Background(),
      datasource.OpenS3DB(), fetchSize, from, to)
   
   source2 := datasource.NewPostgresReader(context.Background(), 
      datasource.OpenPostgresDB(), fetchSize, from, to)
   
   download := NewDownload(from, to)

   start := time.Now()
   print(fmt.Sprintf("downloading trade data from: %s to: %s", 
      from.Format(time.UnixDate), to.Format(time.UnixDate)))
   
   downloadCsvTo(download, "test.csv", source1, source2)
   
   println(" in", time.Since(start).String())

}

func downloadCsvTo(download *Download, file string, source1 io.Reader, source2 io.Reader) {
   fi, err := os.OpenFile(file, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0755)
   if err != nil {
      log.Printf("Error in Create\n")
      panic(err)
   }
   err = download.write(fi, source1, source2)
   if err != nil {
      log.Printf("Error in Create\n")
      panic(err)
   }
   fi.Close()
}


for trying it yourself, please download the git repository, and with docker running and go 1.18 installed

go build, docker-compose up, and in another terminal same folder ./download

(base) ➜  download ./download

populating database ........... splitting data . done

downloading trade data from: Thu Apr 15 14:25:03 CEST 2021 to: Fri Apr 15 15:25:03 CEST 2022 in 310.28149ms

(base) ➜  download 



This is it, a hybrid storage approach applicable under certain circumstances when a lot of immutable data is involved. This is a coarse example, try to add a user/tenant to the data or experiment with buffer sizes