pip-compile: files are not ordered by -c constraints, so layered requirements fail to update #16499

Is there an existing issue for this?

• I have searched the existing issues

Feature description

With pip-tools layered requirements, security updates fail whenever the constrained file sorts before its constraint file alphabetically:

# prod.in -> prod.txt
-r base.in

# dev.in -> dev.txt
-c prod.txt
-r base.in

For a Django bump (5.2.15 → 5.2.17), Dependabot compiles dev.in before prod.in. prod.txt still pins the old version, so the compile fails and the update ends with security_update_not_possible:

pip-compile ... --output-file=dev.txt -P django dev.in    -> exit 1

Cause: order_filenames_for_compilation relies on requirement_map, which only scans -r lines (CHILD_REQUIREMENT_REGEX) and ignores -c (CONSTRAINT_REGEX). Files with no -r link between them are compiled in fetch order, which is alphabetical in practice.

• dependabot-core/python/lib/dependabot/python/file_updater/pip_compile_file_updater.rb

  Lines 753 to 790 in 89a9e44

  	# If the files we need to update require one another then we need to
  	# update them in the right order
  	sig { params(filenames: T::Array[String]).returns(T::Array[String]) }
  	def order_filenames_for_compilation(filenames)
  	ordered_filenames = T.let([], T::Array[String])
  	while (remaining_filenames = filenames - ordered_filenames).any?
  	ordered_filenames +=
  	remaining_filenames
  	.reject do |fn|
  	unupdated_reqs = (requirement_map[fn] || []) - ordered_filenames
  	unupdated_reqs.intersect?(filenames)
  	end
  	end
  	ordered_filenames
  	end
  	sig { returns(T::Hash[String, T::Array[String]]) }
  	def requirement_map
  	child_req_regex = Python::SharedFileFetcher::CHILD_REQUIREMENT_REGEX
  	@requirement_map ||=
  	pip_compile_files.each_with_object({}) do |file, req_map|
  	paths = T.must(file.content).scan(child_req_regex).flatten
  	current_dir = File.dirname(file.name)
  	req_map[file.name] =
  	paths.map do |path|
  	path = File.join(current_dir, path) if current_dir != "."
  	path = Pathname.new(path).cleanpath.to_path
  	path = path.gsub(/\.txt$/, ".in")
  	next if path == file.name
  	path
  	end.uniq.compact
  	end
  	end

• dependabot-core/python/lib/dependabot/python/shared_file_fetcher.rb

  Lines 24 to 25 in 89a9e44

  	CHILD_REQUIREMENT_REGEX = /^-r\s?(?<path>.*\.(?:txt|in))/
  	CONSTRAINT_REGEX = /^-c\s?(?<path>.*\.(?:txt|in))/

This would be solved by also scaning for CONSTRAINT_REGEX in requirement_map, but I'm not sure if that's wanted.

In our case, we renamed the files so alphabetical order matches dependency order (e.g. 1-prod.in, 2-dev.in).

Possibly related: #2272 (same symptom, but that was a fetch issue, fixed in #14588).