feat(storage/transfermanager): prototype #10045
Conversation
sketch of an interface for downloads
product-auto-label
bot
added
the
api: storage
| type Downloader struct { | ||
| client *storage.Client | ||
| config *transferManagerConfig | ||
| work chan *DownloadObjectInput // Piece of work to be executed. |
There was a problem hiding this comment.
Presumably this should be send-only and output should be receive?
There was a problem hiding this comment.
We are sending and receiving from both channels in different places in the downloader. Unidirectional channels could be used in subcomponents or if we were providing the channel to the user, but I don't see how we could implement this with unidirectional channels - if we only received from output, who would send us the output (and vice-versa for work)?
| "google.golang.org/api/iterator" | ||
| ) | ||
|
|
||
| // Downloader manages parallel download operations from a Cloud Storage bucket. |
There was a problem hiding this comment.
Technically bucket can be specified per object. Let's just say that it manages a set of parallelized downloads.
There was a problem hiding this comment.
Makes sense, that wording is clearer.
| return crc32cHash.Sum32(), w.Close() | ||
| } | ||
|
|
||
| type testWriter struct { |
There was a problem hiding this comment.
doesn't have to be in this PR, but we should add a version of this (well, some kind of DownloaderBuffer that implements WriterAt) to the library.
There was a problem hiding this comment.
Yeah, I'd have to look more into that. This is a very barebones implementation that is likely not at all efficient (and doesn't really work as a WriterAt yet).
There was a problem hiding this comment.
Yeah I think that's fine for now.
| } | ||
|
|
||
| // Start workers in background. | ||
| for i := 0; i < d.config.numWorkers; i++ { |
There was a problem hiding this comment.
Presumably we could optimize this by spinning up workers as needed when there are objects enqueued? Doesn't have to be in this PR though.
There was a problem hiding this comment.
Sure, though I'm not sure how much that would optimize this by... I guess it depends on the num of workers.
There was a problem hiding this comment.
Yeah something we can test out later.
BrennaEpp
changed the title
No description provided.