The United States "does not tolerate this kind of ruthless action to silence Mr Khashoggi, a journalist, through violence", Mr Mattis told a meeting in the Bahraini capital Manama.
Turkey alleges a 15-member hit squad was sent to Istanbul to kill the journalist, a onetime Saudi insider who became an outspoken critic of Prince Mohammed in columns for The Washington Post.
US Secretary of State Mike Pompeo this week announced moves against 21 Saudis to either revoke their visas or make them ineligible for US visas after the Khashoggi killing.
Turkish president Recep Tayyip Erdogan made fresh demands on Saudi Arabia to disclose the location of the journalist's body.
He said the kingdom was certain that a "mistake was committed" in the killing of Mr Khashoggi, and that the people responsible had "exceeded their authority".
While these were some of the sharpest comments Mattis has made on the Khashoggi killing, he also said the two countries still needed to collaborate on stability in the region. But his broader mention of the matter toward the end of his speech underscores the serious national security ramifications the incident poses for relations with a key USA ally.
US Defence Secretary James Mattis, who was speaking at the same summit in Bahrain, said more measures would follow.
She also said she had declined an invitation to meet Donald Trump, saying the US President was not honest about investigating the killing.
"This issue has become fairly hysterical", Adel al-Jubeir said at a security forum in Bahrain on Saturday.
"We need to clarify the background of this terrible crime and until that, we will not supply weapons to Saudi Arabia", she added, reiterating comments made earlier this week. The investigating team say they have grilling evidence that will prove this; they say they have some very disturbing audio recordings.
"We are here to demand only one thing from the King and the Crown Prince of the Kingdom of Saudi Arabia: The persons who gave the order for the bestial murder of journalist Jamal Ahmad Khashoggi, and the persons who carried the order, all of them must be brought to account", Nkanga demanded.
"And that is why we are gathered here, to tell the Saudi government that injury to one is injury to all".
Still, Mattis' speech also reflected the hard dilemma this has caused.
Turkey and Saudi Arabia are not known to have an extradition treaty.
Mattis said he continued to support partners in the region who were defending themselves against Houthi attacks in Yemen but also called for an end to fighting there.
Others in the US, however, have condemned the Saudis for what has been called indiscriminate bombings that have slaughtered civilians in Yemen. It was later revealed that the killing was a premeditated murder.
But, he cautioned that respect "must come with transparency and trust".
The developments came after the kingdom on October 25 cited evidence showing Khashoggi's killing was premeditated, changing its story again to try to ease global outrage over the slaying of a prominent critic of Crown Prince Mohammed bin Salman.
More broadly, Mattis' speech on Saturday focused on regional cooperation and the US commitment to the Middle East.
Mattis warned that Russia's presence in the Middle East - where it is a major ally of Syrian President Bashar al-Assad - could not be a replacement for the United States.