[petermyers]

Changes tested by publishing v1.0.0 to locally running Verdaccio and using these test pipelines to verify connectors.

uuid util added to enable mocking since the newer version of the uuid library exports modules in a way that can't be mocked with Sinon.

log util added since the logger interface passable to aws client constructors became slightly more complex and I didn't want to duplicate that logic in every connector.

[jgilbert01]

to maximize v3 modularity, i'm thinking we should remove the various index files and move to using module-based imports.

for example:
import Connector from 'aws-lambda-stream/connectors/dynamodb';

instead of:
import { DynamodbConnector } from 'aws-lambda-stream';

because the index files force the need to include all the dependencies.

we could leave the index files for backward compatibility but deprecate them and stop adding anything to them

thoughts?

[petermyers]

to maximize v3 modularity, i'm thinking we should remove the various index files and move to using module-based imports.

for example: import Connector from 'aws-lambda-stream/connectors/dynamodb';

instead of: import { DynamodbConnector } from 'aws-lambda-stream';

because the index files force the need to include all the dependencies.

we could leave the index files for backward compatibility but deprecate them and stop adding anything to them

thoughts?

Makes sense to me, I can make those changes. I think what I'll do in the vein of compatibility is:

• Include any new exports I added to the index files for this release
• Add deprecation comments to the index files
• Enable module based imports

Then moving forward we can stop adding anything new to them.

[petermyers]

to maximize v3 modularity, i'm thinking we should remove the various index files and move to using module-based imports.
for example: import Connector from 'aws-lambda-stream/connectors/dynamodb';
instead of: import { DynamodbConnector } from 'aws-lambda-stream';
because the index files force the need to include all the dependencies.
we could leave the index files for backward compatibility but deprecate them and stop adding anything to them
thoughts?

Makes sense to me, I can make those changes. I think what I'll do in the vein of compatibility is:

• Include any new exports I added to the index files for this release
• Add deprecation comments to the index files
• Enable module based imports

Then moving forward we can stop adding anything new to them.

You can take a look here for an example of what usage looks like with modular exports. I updated the whole example-connector-test to use the modular exports, so feel free to take a look through that directory in my test repo.

[jgilbert01]

i remove the fetch connector/utils from the index files since they r rarely used

[jgilbert01]

• this major release is also an opportunity to consider breaking the utils folder into a sink folder and a utils
• essentially "through" utils that invoke connectors r sinks

[jgilbert01]

did not mean to close

[petermyers]

• this major release is also an opportunity to consider breaking the utils folder into a sink folder and a utils
• essentially "through" utils that invoke connectors r sinks

I had given that some thought and wanted to pick your brain. I'm generally not a big fan of general 'utils' unless they truly contain utilities like uuid generation, defaults, helpful small utilities to reduce code duplication.

The utils directory as it is now in my mind actually contains 3 things, sinks, through's, and actual utilities. I'm not sold on the name 'throughs', maybe 'stream-utils'? My idea is basically sinks, utilities that operate on the stream (eg. encryption, batching, faults), and basic utilities (eg. retry, opt, print)

My initial thought would be to split the files apart into the following dirs:

• sinks
  • cloudwatch
  • dynamodb
  • eventbridge
  • fetch
  • firehose
  • kinesis
  • lambda
  • s3
  • secretsmgr
  • sns
  • sqs
• through
  • batch
  • claimcheck
  • encryption
  • faults
  • handler
  • ratelimit
  • split
• utils
  • compression
  • log
  • opt
  • print
  • retry
  • tags
  • time
  • uow
  • uuid

Thoughts?

[jgilbert01]

as a first cut to reorganize i created sinks and queries folders and moved utils accordingly

[jgilbert01]

i also removed everything that was deprecated

[jgilbert01]

lots of room to improve on batch get caching

[petermyers]

Definitely. I started here because it mimics how queryAll caching is being done atm. Have some thoughts here, we can discuss. Or if you'd prefer, we can remove this for now and add it in a subsequent release since there is a lot we could potentially do here.

[jgilbert01]

this is a good safe starting position so i'll leave it